Model optimization method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model optimization method, electronic equipment and a storage medium. The method comprises the following steps: acquiring an initial quantization bit width and an initial pruning rate of each convolutional layer in a BEV perception model; on the basis of the initialized quantization bit width of each convolution layer, quantizing the weight of the convolution layer, and according to the initial pruning rate of each convolution layer and the importance score vectors of the multiple output channels, obtaining pruning masks of the output channels; based on the loss function, the quantized weight and the pruning mask training model, obtaining an initial deployment model; and carrying out online quantitative sensitivity evaluation training on the initial deployment model, and when an evaluation condition is met, taking the initial deployment model as a final deployment model. Therefore, the problems that the domain adaptation and generalization ability is limited, the detection precision is reduced, the domain adaptation and generalization ability is difficult to be efficiently utilized by NPU of edge chips such as horizon lines, model evolution in the training process cannot be responded, and error accumulation is serious are solved, and structured sparsity, dynamic adaptation and end-to-end collaborative optimization are achieved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving perception technology, and in particular to a model optimization method, electronic device, and storage medium. Background Technology

[0002] Model compression is crucial for optimizing deep neural network inference on resource-constrained hardware. Among related technologies, View-Guided Distillation (VGD) for Multi-View 3D Object Detection (QD-BEV) addresses the instability of Quantization-Aware Training (QAT) in multi-view 3D detection by proposing a method that optimizes the quantization model using joint image domain and Bird's Eye View (BEV) domain information. Fully Integrated Training Compression (FITCompress) achieves optimal compression of deep neural networks (DNNs) through joint pruning and quantization, adapting to resource-constrained hardware. Another approach integrates pruning and quantization into a differentiable optimization, balancing hardware efficiency and accuracy.

[0003] However, due to the reliance on teacher model accuracy, lack of adaptation to BEV perception characteristics, disruption of the spatial continuity of BEV feature maps during quantization, unstructured pruning, static and fixed compression strategies, and the disconnect between pruning and quantization optimization processes, the related technologies suffer from limited domain adaptation and generalization capabilities, decreased detection accuracy, difficulty in efficient utilization by the neural processing units (NPUs) of edge chips such as Horizon Robotics, inability to respond to model evolution during training, and severe error accumulation. Summary of the Invention

[0004] This application provides a model optimization method, electronic device, and storage medium, aiming to improve the problems of limited domain adaptation and generalization capabilities, decreased detection accuracy, difficulty in efficient utilization of NPUs on edge chips such as Horizon Robotics, inability to respond to model evolution during training, and severe error accumulation, thereby achieving structured sparsity, dynamic adaptation, and end-to-end collaborative optimization.

[0005] To achieve the above objectives, the first aspect of this application proposes a model optimization method, comprising the following steps: Obtain the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model, wherein the BEV perception model includes multiple convolutional layers, and each convolutional layer includes weights and multiple output channels; Based on the initial quantization bit width of each convolutional layer, the weights of the corresponding convolutional layer are quantized to obtain the quantized weights of each convolutional layer. The pruning mask of the output channel of the corresponding convolutional layer is obtained according to the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. Based on the loss function of the BEV perception model, the quantized weights of each convolutional layer, and the pruning mask of the output channel, the BEV perception model is trained until the preset training termination condition is met to obtain the initial deployment model. The initial deployment model is trained using an online quantitative sensitivity assessment to obtain the assessment results. When the assessment results meet the preset assessment conditions, the initial deployment model is used as the final deployment model.

[0006] Therefore, the initial quantization bit width and initial pruning rate of each convolutional layer in the BEV perception model are obtained. The weights of the convolutional layers are quantized based on the initial quantization bit width. The pruning mask of the output channel is obtained according to the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. The model is then trained based on the loss function, quantized weights, and pruning mask to obtain the initial deployment model. Online quantization sensitivity evaluation training is performed on the initial deployment model. When the evaluation conditions are met, the initial deployment model is used as the final deployment model. This solves the problems of limited domain adaptation and generalization capabilities, decreased detection accuracy, difficulty in efficient utilization of the NPU of edge chips such as Horizon Robotics, inability to respond to model evolution during training, and severe error accumulation. It achieves structured sparsity, dynamic adaptation, and end-to-end collaborative optimization.

[0007] According to one embodiment of this application, obtaining the pruning mask for the output channel of the corresponding convolutional layer based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels includes: Determine the importance score vector for each output channel of each of the convolutional layers; Based on the importance score vector of each output channel of each convolutional layer, the multiple output channels of each convolutional layer are sorted in descending order to obtain the sorting result of each convolutional layer; Based on the sorting result of each convolutional layer, the output channels of the corresponding convolutional layer are grouped to obtain multiple group channels for each convolutional layer. Calculate the average importance score vector for each group of channels in each of the convolutional layers; The pruning mask for the output channel of each convolutional layer is obtained based on the initial pruning rate of each convolutional layer and the average importance score vector of the group channels.

[0008] Therefore, by adopting a pruning strategy of channel grouping and group average importance scoring, redundant channels are accurately identified, ensuring that the channel structure is regular after pruning to adapt to hardware requirements, and achieving a balance between model lightweighting and perception accuracy.

[0009] According to one embodiment of this application, obtaining the pruning mask for the output channel of each convolutional layer based on the initial pruning rate of each convolutional layer and the average importance score vector of the group channels includes: Based on the importance score vector of the output channel of each convolutional layer, calculate the average importance score vector of the group channels of the corresponding convolutional layer; Based on the average importance score vector of the group channels of each convolutional layer, target groups whose average importance score vector is greater than or equal to the initial pruning rate of the corresponding convolutional layer are selected to obtain the target groups of each convolutional layer; Determine the group mask of the target group for each convolutional layer, and obtain the pruning mask for each output channel of each convolutional layer based on the group mask of the target group for each convolutional layer.

[0010] Therefore, by selecting target groups through average group scores and generating pruning masks, the retention channels can be accurately locked, simplifying the pruning decision-making process and ensuring the model's lightweight nature and hardware compatibility.

[0011] According to one embodiment of this application, the step of quantizing the weights of the corresponding convolutional layers based on the initial quantization bit width of each convolutional layer to obtain the quantized weights of each convolutional layer includes: Based on the initial quantization bit width of each convolutional layer, pseudo-quantization nodes are inserted into the corresponding convolutional layer weights and activation values ​​during the forward pass, and quantization-dequantization is performed to generate the quantized weights of each convolutional layer.

[0012] This helps retain key information about the weights, improves the reliability of the quantized weights, and provides a stable foundation for subsequent training and deployment.

[0013] According to one embodiment of this application, the step of performing online quantitative sensitivity evaluation training on the initial deployment model to obtain evaluation results, and using the initial deployment model as the final deployment model when the evaluation results meet preset evaluation conditions, includes: Based on a preset sliding window, an inference is performed on the initial deployment model once according to a preset verification period and a preset verification set to obtain multiple dequantized feature maps of each convolutional layer within the preset sliding window; The feature similarity of each convolutional layer within the preset sliding window is calculated based on the multiple dequantized feature maps of each convolutional layer within the preset sliding window. Multiple sensitivities of each convolutional layer within the preset sliding window are obtained based on these multiple feature similarities. The average sensitivity of each convolutional layer is then calculated based on these multiple sensitivities. The sensitivity represents the degree to which the feature perturbation caused by the quantization operation of the corresponding convolutional layer affects the final accuracy of the perception task. Higher sensitivity requires a higher bit width for the convolutional layer, while lower sensitivity allows for a lower bit width. Based on the threshold buffer where the average sensitivity of each convolutional layer is located, the target quantization bit width of each convolutional layer is determined, and the model is trained according to the target quantization bit width of each convolutional layer until the maximum number of training iterations is reached, thus obtaining the final deployment model.

[0014] Therefore, by using a sliding window combined with a feature similarity-based quantization sensitivity evaluation method, the degree of feature perturbation after quantization of each convolutional layer can be accurately measured. Based on this, the quantization bit width of the differentiated target can be matched, achieving efficient and lightweight optimization while ensuring the model's perception accuracy.

[0015] According to one embodiment of this application, the loss function of the BEV perception model includes a prediction loss term and at least one regularization term, wherein the at least one regularization term includes a pruning regularization term, a quantization perception regularization term, and a spatial gradient regularization term, wherein... The pruning regularization term is configured to apply a sparse penalty to each output channel vector of each of the convolutional layers to drive the score of unimportant channels to monotonically decrease to zero during training. The quantization-aware regularization term is configured to establish a differentiable pass-through estimation difference measure between the floating-point weights and the quantized integer values; The spatial gradient regularization term is configured to apply a smoothing penalty to the first-order gradient magnitudes of the BEV feature map in the horizontal and vertical directions to reduce spatial discontinuities caused by quantization noise.

[0016] Therefore, by designing a composite loss function that integrates prediction loss terms and multi-dimensional regularization terms, we can achieve synergistic optimization of channel sparsity, quantization error suppression and BEV feature space smoothness, while simultaneously taking into account the model's perception accuracy and lightweight deployment requirements during training.

[0017] According to one embodiment of this application, obtaining the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model includes: Calculate the quantization sensitivity of each convolutional layer in the BEV perception model; The initial quantization bit width and initial pruning rate of each convolutional layer are obtained by initializing the quantization sensitivity of each convolutional layer.

[0018] Therefore, by determining the initial quantization bit width and initial pruning rate, the initial parameters are adapted to the differences in inter-layer characteristics, avoiding compression imbalance caused by blind setting, laying an accurate foundation for subsequent training optimization, and improving model compression efficiency and performance retention.

[0019] According to one embodiment of this application, calculating the quantization sensitivity of each convolutional layer in the BEV perception model includes: Based on a preset validation set, the BEV perception model is subjected to complete inference, and the original task loss of each convolutional layer is obtained based on the inference results. The target convolutional layer is determined from the BEV perception model, and the target convolutional layer is quantized using INT8 while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model. The new BEV perception model is then subjected to full inference using the preset validation set, and the quantization task loss of the target convolutional layer is obtained based on the inference result. A new convolutional layer is determined from the remaining convolutional layers of the BEV perception model, and the new convolutional layer is used as the target convolutional layer. The step of performing INT8 quantization on the target convolutional layer is re-executed while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model, until the quantization task loss of each convolutional layer is obtained. The quantization sensitivity increment of each convolutional layer is calculated based on the quantization task loss and the original task loss of each convolutional layer, and the quantization sensitivity increment of each convolutional layer is normalized to obtain the quantization sensitivity of each convolutional layer.

[0020] Therefore, it can accurately capture the differences in sensitivity to quantization at each layer, and the evaluation results are directly related to task performance, providing a reliable basis for the reasonable setting of the initial quantization bit width and pruning rate.

[0021] According to the model optimization method proposed in this application, the initial quantization bit width and initial pruning rate of each convolutional layer in the BEV perception model are obtained. The weights of the convolutional layers are quantized based on the initial quantization bit width. The pruning mask of the output channel is obtained based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. The model is then trained based on the loss function, quantized weights, and pruning mask to obtain the initial deployment model. The initial deployment model is then trained with online quantization sensitivity evaluation. When the evaluation conditions are met, the initial deployment model is used as the final deployment model. This solves the problems of limited domain adaptation and generalization capabilities, decreased detection accuracy, difficulty in efficient utilization of the NPU of edge chips such as Horizon Robotics, inability to respond to model evolution during training, and severe error accumulation. It achieves structured sparsity, dynamic adaptation, and end-to-end collaborative optimization.

[0022] To achieve the above objectives, a second aspect of this application provides a model optimization apparatus, comprising: The acquisition module acquires the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model, wherein the BEV perception model includes multiple convolutional layers, and each convolutional layer includes weights and multiple output channels. The quantization module quantizes the weights of each convolutional layer based on the initial quantization bit width of each convolutional layer to obtain the quantized weights of each convolutional layer, and obtains the pruning mask of the output channel of the corresponding convolutional layer based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. The training module trains the BEV perception model based on the loss function of the BEV perception model, the quantized weights of each convolutional layer, and the pruning mask of the output channel until a preset training termination condition is met, so as to obtain the initial deployment model. The optimization module performs online quantitative sensitivity evaluation training on the initial deployment model to obtain evaluation results, and when the evaluation results meet preset evaluation conditions, the initial deployment model is used as the final deployment model.

[0023] According to one embodiment of this application, the quantization module is specifically used for: Determine the importance score vector for each output channel of each of the convolutional layers; Based on the importance score vector of each output channel of each convolutional layer, the multiple output channels of each convolutional layer are sorted in descending order to obtain the sorting result of each convolutional layer; Based on the sorting result of each convolutional layer, the output channels of the corresponding convolutional layer are grouped to obtain multiple group channels for each convolutional layer. Calculate the average importance score vector for each group of channels in each of the convolutional layers; The pruning mask for the output channel of each convolutional layer is obtained based on the initial pruning rate of each convolutional layer and the average importance score vector of the group channels.

[0024] According to one embodiment of this application, the quantization module is specifically used for: Based on the importance score vector of the output channel of each convolutional layer, calculate the average importance score vector of the group channels of the corresponding convolutional layer; Based on the average importance score vector of the group channels of each convolutional layer, target groups whose average importance score vector is greater than or equal to the initial pruning rate of the corresponding convolutional layer are selected to obtain the target groups of each convolutional layer; Determine the group mask of the target group for each convolutional layer, and obtain the pruning mask for each output channel of each convolutional layer based on the group mask of the target group for each convolutional layer.

[0025] According to one embodiment of this application, the quantization module is specifically used for: Based on the initial quantization bit width of each convolutional layer, pseudo-quantization nodes are inserted into the corresponding convolutional layer weights and activation values ​​during the forward pass, and quantization-dequantization is performed to generate the quantized weights of each convolutional layer.

[0026] According to one embodiment of this application, the optimization module is specifically used for: Based on a preset sliding window, an inference is performed on the initial deployment model once according to a preset verification period and a preset verification set to obtain multiple dequantized feature maps of each convolutional layer within the preset sliding window; The feature similarity of each convolutional layer within the preset sliding window is calculated based on the multiple dequantized feature maps of each convolutional layer within the preset sliding window. Multiple sensitivities of each convolutional layer within the preset sliding window are obtained based on these multiple feature similarities. The average sensitivity of each convolutional layer is then calculated based on these multiple sensitivities. The sensitivity represents the degree to which the feature perturbation caused by the quantization operation of the corresponding convolutional layer affects the final accuracy of the perception task. Higher sensitivity requires a higher bit width for the convolutional layer, while lower sensitivity allows for a lower bit width. Based on the threshold buffer where the average sensitivity of each convolutional layer is located, the target quantization bit width of each convolutional layer is determined, and the model is trained according to the target quantization bit width of each convolutional layer until the maximum number of training iterations is reached, thus obtaining the final deployment model.

[0027] According to one embodiment of this application, the loss function of the BEV perception model includes a prediction loss term and at least one regularization term, wherein the at least one regularization term includes a pruning regularization term, a quantization perception regularization term, and a spatial gradient regularization term, wherein... The pruning regularization term is configured to apply a sparse penalty to each output channel vector of each of the convolutional layers to drive the score of unimportant channels to monotonically decrease to zero during training. The quantization-aware regularization term is configured to establish a differentiable pass-through estimation difference measure between the floating-point weights and the quantized integer values; The spatial gradient regularization term is configured to apply a smoothing penalty to the first-order gradient magnitudes of the BEV feature map in the horizontal and vertical directions to reduce spatial discontinuities caused by quantization noise.

[0028] According to one embodiment of this application, the acquisition module is specifically used for: Calculate the quantization sensitivity of each convolutional layer in the BEV perception model; The initial quantization bit width and initial pruning rate of each convolutional layer are obtained by initializing the quantization sensitivity of each convolutional layer.

[0029] According to one embodiment of this application, the acquisition module is specifically used for: Based on a preset validation set, the BEV perception model is subjected to complete inference, and the original task loss of each convolutional layer is obtained based on the inference results. The target convolutional layer is determined from the BEV perception model, and the target convolutional layer is quantized using INT8 while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model. The new BEV perception model is then subjected to full inference using the preset validation set, and the quantization task loss of the target convolutional layer is obtained based on the inference result. A new convolutional layer is determined from the remaining convolutional layers of the BEV perception model, and the new convolutional layer is used as the target convolutional layer. The step of performing INT8 quantization on the target convolutional layer is re-executed while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model, until the quantization task loss of each convolutional layer is obtained. The quantization sensitivity increment of each convolutional layer is calculated based on the quantization task loss and the original task loss of each convolutional layer, and the quantization sensitivity increment of each convolutional layer is normalized to obtain the quantization sensitivity of each convolutional layer.

[0030] According to the model optimization apparatus proposed in this application, the initial quantization bit width and initial pruning rate of each convolutional layer in the BEV perception model are obtained. The weights of the convolutional layers are quantized based on the initial quantization bit width. The pruning mask of the output channel is obtained based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. The model is trained based on the loss function, quantized weights, and pruning mask to obtain an initial deployment model. The initial deployment model is then trained with online quantization sensitivity evaluation. When the evaluation conditions are met, the initial deployment model is used as the final deployment model. This solves the problems of limited domain adaptation and generalization capabilities, decreased detection accuracy, difficulty in efficient utilization of the NPU of edge chips such as Horizon Robotics, inability to respond to model evolution during training, and severe error accumulation. It achieves structured sparsity, dynamic adaptation, and end-to-end collaborative optimization.

[0031] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the model optimization method as described in the above embodiments.

[0032] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the model optimization method as described in the above embodiments.

[0033] To achieve the above objectives, a fifth aspect of this application provides a computer program product, which, when executed by a processor, implements the model optimization method as described in the above embodiments.

[0034] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0035] Figure 1 This is a flowchart of the model optimization method provided according to the embodiments of this application; Figure 2 This is a flowchart of a model optimization method provided according to an embodiment of this application; Figure 3 This is a block diagram of a model optimization apparatus provided according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0036] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] The model optimization method, electronic device, and storage medium proposed according to embodiments of this application are described below with reference to the accompanying drawings.

[0038] Figure 1 This is a flowchart of a model optimization method according to an embodiment of this application.

[0039] like Figure 1 As shown, the model optimization method includes the following steps: In step S101, the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model are obtained. The BEV perception model includes multiple convolutional layers, and each convolutional layer includes weights and multiple output channels.

[0040] In some embodiments, obtaining the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model includes: calculating the quantization sensitivity of each convolutional layer in the BEV perception model; and initializing the quantization bit width and initial pruning rate of the corresponding convolutional layer based on the quantization sensitivity of each convolutional layer to obtain the initial quantization bit width and initial pruning rate of each convolutional layer.

[0041] In some embodiments, calculating the quantization sensitivity of each convolutional layer in the BEV perception model includes: performing full inference on the BEV perception model based on a preset validation set, and obtaining the original task loss of each convolutional layer based on the inference results; determining a target convolutional layer from the BEV perception model, performing INT8 quantization on the target convolutional layer, and maintaining the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model, and performing full inference on the new BEV perception model using a preset validation set, and obtaining the quantization task loss of the target convolutional layer based on the inference results; determining a new convolutional layer from the remaining convolutional layers of the BEV perception model, using the new convolutional layer as the target convolutional layer, and re-performing the steps of performing INT8 quantization on the target convolutional layer, and maintaining the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model, until the quantization task loss of each convolutional layer is obtained; calculating the quantization sensitivity increment of each convolutional layer based on the quantization task loss of each convolutional layer and the original task loss of each convolutional layer, and normalizing the quantization sensitivity increment of each convolutional layer to obtain the quantization sensitivity of each convolutional layer.

[0042] Quantization sensitivity refers to the amplification effect on the overall task loss when a single convolutional layer is reduced from full precision to low bit depth. Quantization bit width refers to the number of bits occupied by the discretized weights or activation values ​​of a single convolutional layer. Pruning rate refers to the proportion of permanently zeroed weights or output channels in a single convolutional layer to the total weights or channels of that layer. The preset validation set can be a user-defined validation set, a validation set obtained through a finite number of experiments, or a validation set obtained through a finite number of computer simulations. Full inference refers to performing a complete forward propagation computation from start to finish on the model using the preset validation set to obtain the model's final task output on all samples. The original task loss refers to the loss value calculated using the task-related loss function after performing full inference based on the preset validation set, with all convolutional layers maintaining their original high-precision format and without any quantization processing. Target layer convolution refers to the specific convolutional layer selected sequentially as the target layer in the quantization sensitivity evaluation process. The remaining convolutional layers retain their original high-precision format. The selection strategy is to traverse layer by layer sequentially, activating only one convolutional layer at a time according to the network topology (or reverse order) until all layers to be analyzed are covered. Random selection is not allowed, nor are multiple layers activated simultaneously. INT8 quantization (8-bit Integer Quantization) refers to the discretization process of mapping convolutional layer weights and activation values ​​from high-bit-width floating-point numbers to 8-bit integers. This process reduces the overall precision of network parameters and intermediate features, resulting in a simultaneous decrease in model size, memory bandwidth, and computational power consumption. However, it introduces quantization noise and may cause a loss of task precision. Besides INT8 quantization, there are other integer quantization methods with different bit widths, such as INT4, INT6, and INT2. Any method that converts high-precision floating-point parameters into low-precision integer formats through mapping to achieve model lightweighting falls under the category of integer quantization.

[0043] Specifically, this embodiment uses the open-source BEVDet (Bird's Eye View Detection) model as an FP32 (32-bit Floating Point) pre-trained model (i.e., the BEV perception model). Based on this, the initial quantization bit width and initial pruning rate of each convolutional layer in the BEV perception model are calculated: the initial quantization bit width is used to define the initial precision threshold for converting weights from high-precision floating-point numbers to low-precision formats, and the initial pruning rate is used to determine the initial sparsity ratio of each layer's channels.

[0044] Furthermore, based on a pre-defined validation set, a complete inference process is performed on the BEV perception model. The original task loss, denoted as L, is calculated using a loss function. fpThis loss value serves as the baseline loss for all convolutional layers in their unquantized states. For any convolutional layer in the BEV perception model, let's denote it as the _th_ convolutional layer. i Layer convolution, the first i The output of the convolution layer is For the first in the model i Each convolutional layer is individually quantized using INT8, while the remaining convolutional layers maintain a high-precision format (e.g., FP32), resulting in a new BEV perception model. Using a pre-defined validation set, the new BEV perception model is subjected to complete inference. The INT8 model is then calculated using a loss function to obtain the result for only the first convolutional layer. i The task loss of the model after INT8 quantization of the convolutional layers is denoted as L. i . No. i The formula for calculating the quantization sensitivity increment of a convolutional layer is: ; in, For the first i The quantization sensitivity increment of the convolutional layer, For only the first i The task loss of the model after INT8 quantization of convolutional layers. This represents the loss of the original task.

[0045] For the i The quantization sensitivity increment of the convolutional layer is normalized to obtain the quantization sensitivity, calculated using the following formula: ; in, For the first i Quantization sensitivity of convolutional layers; This is the maximum value among the quantization sensitivity increments of all convolutional layers; It is the minimum value among all the quantization sensitivity increments of all convolutional layers.

[0046] A larger value indicates that the layer is more sensitive to quantization.

[0047] The quantization sensitivity based on the i-th convolutional layer is Initialize the first i Quantization bit width of layer convolution and initial pruning rate The initial formula is as follows: ; ; in, To quantize the common threshold for bit width and pruning rate, This represents the high threshold for the pruning rate.

[0048] Further, following the above process, the next convolutional layer that has not yet undergone quantization testing is selected from the BEV perception model as the new target convolutional layer. INT8 quantization is performed on this layer individually (the remaining convolutional layers maintain their original high-precision format), resulting in an updated BEV perception model. A pre-defined validation set is used to perform complete inference on the updated BEV perception model, and the quantized task loss of this layer is calculated and recorded. The steps of selecting untested convolutional layers, quantizing these layers individually, and inferring and recording the corresponding task loss are repeated until all convolutional layers in the BEV perception model have completed independent quantization testing, yielding the quantization task loss for each convolutional layer. Based on the quantization task loss and the original task loss of each convolutional layer, the quantization sensitivity increment of each convolutional layer is calculated. Finally, the quantization sensitivity increments of all layers are normalized to obtain the quantization sensitivity of each convolutional layer.

[0049] In step S102, the weights of the corresponding convolutional layers are quantized based on the initial quantization bit width of each convolutional layer to obtain the quantized weights of each convolutional layer, and the pruning masks of the output channels of the corresponding convolutional layers are obtained according to the initial pruning rate of each convolutional layer and the importance score vectors of multiple output channels.

[0050] Optionally, in some embodiments, the weights of the corresponding convolutional layers are quantized based on the initial quantization bit width of each convolutional layer to obtain the quantized weights of each convolutional layer, including: inserting pseudo-quantization nodes into the corresponding convolutional layer weights and activation values ​​during the forward process based on the initial quantization bit width of each convolutional layer, and performing quantization-dequantization to generate the quantized weights of each convolutional layer.

[0051] In this context, the weights of a convolutional layer refer to the learnable parameters within the convolutional layers of a BEV-aware model used for feature extraction, typically existing as two-dimensional matrices or three-dimensional tensors. The corresponding layer weights refer to the learnable parameter matrix of a specific convolutional layer directly associated with the current operation. Activation values ​​refer to the intermediate output features generated by the convolutional layers in a neural network when processing input data. Pseudo-quantized nodes refer to simulated quantization units inserted into the computational path of network layers during quantization-aware training.

[0052] Specifically, in this embodiment of the application, the model is trained using quantization awareness. During the training process, pseudo-quantization nodes are used to simulate the quantization process. Regarding the first... i Convolutional layers, based on the current quantization bit width b iIn the forward propagation computation path, pseudo-quantization nodes are inserted for the weights and activation values ​​of this layer. The core parameters of the pseudo-quantization nodes are determined by the initial quantization bit width of the layer. During the forward propagation, the pseudo-quantization nodes first perform quantization, mapping the floating-point weights and activation values ​​to the integer quantization space (e.g., FP32-INT8) using a mapping formula; then, they perform dequantization, converting the integers back to the floating-point space (e.g., INT8-FP32) using a reverse mapping, and then calculate the gradient based on the floating-point values, ensuring the continuity and differentiability of parameter updates. The mapping formula is as follows: ; in, The quantized integer value; (·) represents the rounding operation; x The floating-point value to be quantized; scale This is a scaling factor used to map the range of floating-point values ​​to the integer range of the quantization bit width; Zero is used to align floating-point "0" with the quantized integer "0".

[0053] Because the discretization operation in the quantization process is discontinuous and lacks differentiability at the mathematical level, it hinders gradient calculation in the backpropagation stage. To address this issue, this application employs a Straight-Through Estimator (STE) to approximate the gradient. Specifically, the quantization operation is performed during the forward propagation stage, while the influence of the quantization operation is ignored during backpropagation, and the gradient is directly transmitted.

[0054] Further, in some embodiments, the pruning mask for the output channel of the corresponding convolutional layer is obtained based on the initial pruning rate of each convolutional layer and the importance score vectors of multiple output channels, including: determining the importance score vector of each output channel of each convolutional layer; sorting the multiple output channels of each convolutional layer in descending order based on the importance score vector of each output channel of each convolutional layer to obtain the sorting result of each convolutional layer; dividing the output channels of the corresponding convolutional layer into groups according to the sorting result of each convolutional layer to obtain multiple group channels of each convolutional layer; calculating the average importance score vector of the group channels of each convolutional layer; and obtaining the pruning mask for the output channel of each convolutional layer based on the initial pruning rate of each convolutional layer and the average importance score vector of the group channels.

[0055] Optionally, in some embodiments, obtaining the pruning mask for the output channel of each convolutional layer based on the initial pruning rate and the average importance score vector of the group channels of each convolutional layer includes: calculating the average importance score vector of the group channels of the corresponding convolutional layer based on the importance score vector of the output channels of each convolutional layer; selecting target groups whose average importance score vector is greater than or equal to the initial pruning rate of the corresponding convolutional layer based on the average importance score vector of the group channels of each convolutional layer, thereby obtaining the target groups of each convolutional layer; determining the group mask of the target groups of each convolutional layer, and obtaining the pruning mask for each output channel of each convolutional layer based on the group mask of the target groups of each convolutional layer.

[0056] The importance score vector for multiple output channels refers to a one-dimensional numerical vector calculated for all output channels of a single convolutional layer to measure the contribution of each channel to the task. Its dimension is consistent with the total number of output channels of the convolutional layer. Each element in the vector corresponds to the importance score of an output channel, and the score indicates the degree of contribution of the corresponding channel to the model's perception task. The average importance score vector for each group channel of the convolutional layer refers to a one-dimensional vector generated in the BEV perception model convolutional layer channel pruning process. First, the sorted output channels of the convolutional layer are divided into several groups (i.e., group channels). Then, the importance scores of all output channels in each group are averaged. Finally, an average score value is generated for each group channel and arranged in the order of the groups to form a one-dimensional vector. Its dimension is consistent with the number of group channels of the convolutional layer, providing a unified and simplified basis for importance judgment for group-level structured pruning.

[0057] Specifically, a learnable channel importance score vector is introduced for each convolutional layer, such as the channel importance score vector for the i-th convolutional layer. , The initial values ​​are all 1. The number of output channels indicates the importance of each output channel.

[0058] When the number of training epochs exceeds a preset threshold N0, a channel grouping and pruning process is initiated. For each convolutional layer in the model, the C output channels of that layer are sorted in descending order according to their channel importance score vectors. The sorted C channels are then divided into G groups, where the number of groups G is determined by the hardware alignment width v. Prioritizing setting C to an integer multiple of v maximizes hardware inference efficiency. If C is not an integer multiple of v, the total number of channels can be adjusted to an integer multiple of v by adding redundant channels or pruning a small number of low-importance channels before grouping. For example, if v=8, and C satisfies the requirement of 8 times the number of hardware channels for horizon alignment, then the number of groups... .

[0059] Taking the j-th group as an example, for G j, i.e., the group number of the j-th group, calculate the average importance score within the group using the following formula: ; in, The average importance score for group j, Score the importance of the k-th channel within the group.

[0060] The average importance score for each group With the initial pruning threshold Compare, if If the condition is met, the group is kept; otherwise, the group is removed. Group mask for: ; Based on the index of the group to which the channel belongs and the group mask, the channel-level pruning mask for each convolutional layer is obtained. The pruning mask for the i-th convolutional layer is: This indicates whether each channel is retained; 1 means retained, and 0 means deleted.

[0061] Furthermore, an initial pruning threshold is calculated for each convolutional layer based on its initial pruning rate. To ensure the initial threshold is reasonable, the quantization sensitivity set S of all convolutional layers in the model is sorted in descending order. Select the sorted number Percentile values ​​are used as the initial pruning threshold. pruning threshold The initialization formula is: ; in, To select the first element after sorting set S in descending order. The percentile value.

[0062] As training progresses, the pruning threshold is updated using the following formula: ; in, To update the step size; The minimum pruning threshold; The maximum pruning threshold; ·) is the truncation function, ensuring that the updated threshold does not exceed [ , ].

[0063] In step S103, the BEV perception model is trained based on the loss function of the BEV perception model, the quantized weights of each convolutional layer, and the pruning mask of the output channel until the preset training termination condition is met, so as to obtain the initial deployment model.

[0064] Optionally, in some embodiments, the loss function of the BEV-aware model includes a prediction loss term and at least one regularization term. The at least one regularization term includes a pruning regularization term, a quantization-aware regularization term, and a spatial gradient regularization term. The pruning regularization term is configured to impose a sparsity penalty on each output channel vector of each convolutional layer to drive the scores of unimportant channels to monotonically decrease to zero during training. The quantization-aware regularization term is configured to establish a differentiable pass-through estimation difference measure between floating-point weights and quantized integer values. The spatial gradient regularization term is configured to impose a smoothing penalty on the first-order gradient magnitudes of the BEV feature map in the horizontal and vertical directions to reduce spatial discontinuities caused by quantization noise.

[0065] The loss function of the BEV perception model is a quantization function used to measure the difference between the BEV perception model's prediction results and the true labels. It also integrates regularization constraints related to pruning, quantization, and feature smoothing, and is the core optimization objective in the joint pruning and quantization training process. Its value directly reflects the model's perception accuracy and compression adaptability. The preset training termination conditions can be user-defined, obtained through a limited number of experiments, or derived from a limited number of computer simulations.

[0066] Specifically, in the joint pruning and quantization-aware training, the pruning regularization term guides low-importance channels to actively exit through regularization constraints, forming a collaborative optimization with quantization-aware training. The pruning regularization term is as follows: ; in, The pruning regularization loss function; It is the L1 norm, which is the sum of the absolute values ​​of all elements of the vector.

[0067] During training, the scores of unimportant channels are pushed close to zero, naturally sparsifying the parameters of low-importance channels and achieving automated structured pruning. This regularization term can be co-optimized with quantization-aware training to ensure that the model maintains high accuracy performance in the BEV perception task during the lightweight compression process.

[0068] Furthermore, to promote the convergence of floating-point weights to integer values ​​after vectorization during training and reduce parameter information loss during quantization deployment, this application introduces a quantization-aware regularization term: ; in, To quantize the perceptual regularization loss function, These are the original floating-point weights of the i-th convolutional layer. The weights of the i-th layer convolution at the current quantization bit width The pseudo-quantization results below It is the square of the L2 norm.

[0069] Since quantization is a discrete and discontinuous process, it is mathematically nondifferentiable and cannot be calculated directly. right The gradient of the can cause gradient interruption, so this regularization term enables gradient backpropagation through a pass-through estimator, guiding the weights to converge toward a representable discrete value.

[0070] Furthermore, to address the issue that quantization noise can easily disrupt the spatial continuity of feature maps in BEV perception tasks, a spatial gradient regularization term is introduced: ; in, Let be the spatial gradient regularization loss function. These are the feature values ​​of the quantized BEV feature map at width x and height y. This is the quantized BEV feature map, where X and Y are the width and height of the feature map, respectively.

[0071] The spatial gradient regularization term is applied to the BEV feature map to suppress high-frequency spatial noise introduced by quantization, thus maintaining the clarity of object boundaries and the corresponding intensity of distant targets.

[0072] The final loss function, i.e., the preset loss function, takes into account accuracy loss, pruning rate, and quantization error simultaneously, and is calculated using the following formula: ; in, For the final loss function, This is the loss function for BEV target detection tasks, which depends on the specific task.

[0073] In step S104, the initial deployment model is trained with online quantization sensitivity assessment to obtain the assessment results. When the assessment results meet the preset assessment conditions, the initial deployment model is used as the final deployment model.

[0074] Optionally, in some embodiments, the initial deployment model is trained using online quantization sensitivity evaluation to obtain evaluation results. When the evaluation results meet preset evaluation conditions, the initial deployment model is used as the final deployment model. This includes: performing one inference operation on the initial deployment model based on a preset sliding window, according to a preset verification period and a preset verification set, to obtain multiple dequantized feature maps of each convolutional layer within the preset sliding window; calculating the feature similarity of each convolutional layer within the preset sliding window based on the multiple dequantized feature maps of each convolutional layer within the preset sliding window; and calculating the feature similarity of each convolutional layer within the preset sliding window based on the feature similarities of each convolutional layer within the preset sliding window. Multiple feature similarities are used to obtain multiple sensitivities for each convolutional layer within a preset sliding window. The average sensitivity of each convolutional layer is calculated based on the multiple sensitivities of each convolutional layer. Sensitivity is used to represent the degree of influence of feature perturbation caused by quantization operation on the final perception task accuracy of the corresponding convolutional layer. The higher the sensitivity, the higher the bit width that the convolutional layer needs to retain, and vice versa. The target quantization bit width of each convolutional layer is determined based on the threshold buffer where the average sensitivity of each convolutional layer is located. The model is trained based on the target quantization bit width of each convolutional layer until the maximum number of training iterations is reached, and the final deployment model is obtained.

[0075] The preset evaluation conditions can be user-defined conditions, conditions obtained through a limited number of experiments, or conditions obtained through a limited number of computer simulations. Feature similarity refers to a quantitative metric that measures the degree of similarity between multiple dequantized feature maps of the same convolutional layer within a preset sliding window. Target quantization bit width refers to the number of bits used to represent the final low-precision values ​​for each convolutional layer during the model quantization deployment phase.

[0076] Furthermore, an online quantization sensitivity assessment is performed on the initially deployed model. After the number of training epochs exceeds a preset threshold N0, every N1 (where N1 is a positive integer and satisfies...)... After each training round, online sensitivity updates are performed and the quantization bit width and pruning rate are adjusted.

[0077] The online sensitivity calculation method is as follows: In the current training state, inference operations are performed using a preset validation set, and the output features of the i-th layer of convolution dequantization in the INT low-precision model are recorded. The formula for calculating the feature similarity of the i-th convolutional layer is as follows: ; in, Let be the feature similarity of the i-th convolutional layer. Let be the feature map with 32-bit floating-point precision of the i-th convolutional layer. This is the feature map after inverse quantization of the i-th convolutional layer.

[0078] The sensitivity is obtained by using the feature similarity of the i-th convolutional layer, and the calculation formula is as follows: ; Furthermore, this application introduces a sliding window average sensitivity, which is a smoothed sensitivity calculation index designed for online quantization sensitivity evaluation of BEV perception models. Its core is to set a sliding time window of fixed length w (corresponding to w consecutive online sensitivity update records), and to perform an average calculation on the w single online sensitivity values ​​generated by the i-th convolutional layer in the initial deployment model within the window to obtain the sliding window average sensitivity of the convolutional layer. Its function is to filter out random fluctuations and outliers in single online sensitivity calculations, avoid extreme impacts of single sensitivity results on quantization bit width adjustment, and improve the stability and reliability of quantization bit width decisions.

[0079] To avoid excessive influence of a single online sensitivity calculation result on the quantization bit width decision, a sliding window of length w is adopted for the first... i The sensitivity of each convolutional layer is calculated by averaging, as follows: ; in, e For the current training round, The average sensitivity of the i-th layer convolution in the current round. Let be the sensitivity of the i-th convolutional layer in the t-th round.

[0080] Furthermore, this method effectively smooths short-term sensitivity fluctuations, ensuring that quantization bit width adjustments are driven by long-term trends. Simultaneously, to avoid frequent quantization bit width switching, a threshold buffer [T] is specifically set. m ,T high When the average sensitivity of the convolutional layer is within this range, the quantization bit width remains unchanged. i The target quantization bit width for each convolutional layer is: ; Model training is conducted based on the target quantization bit width of each convolutional layer, and training is terminated when the maximum number of training epochs (MaxEpoch) is reached. The weights and pruning mask of the epoch with the highest Mean Average Precision (mAP) on the test set are selected and compiled to generate the final deployment model that can be used for inference on Horizon Robotics chips.

[0081] Therefore, by complementing offline and online sensitivity assessments—that is, by integrating offline and online quantization sensitivity assessments into a two-stage dynamic perceptron—the impact of global tasks and the preservation of local features are balanced, enabling dynamic adaptive adjustment of the compression strategy and reducing reliance on manual parameter tuning. A structured grouping pruning mechanism with 8-channel aligned Horizon computing units is adopted, employing a learnable structured grouping pruning mechanism for Horizon hardware acceleration, compatible with hardware acceleration and reducing memory fragmentation and scheduling overhead. High precision and structural integrity are maintained at critical layers, while non-critical layers are significantly compressed. The Horizon chip's INT8 / INT16 mixed precision capability is fully utilized to avoid full INT8 precision collapse. Joint optimization of pruning and quantization breaks error accumulation, adapting to the feature space continuity and view transformation sensitivity of BEV perception. This approach is superior to traditional pruning-then-quantization or quantization-then-pruning schemes, avoiding error aggregation.

[0082] To facilitate a better understanding of the model optimization method in the embodiments of this application by those skilled in the art, the following is combined with... Figure 2 Further explanation is needed.

[0083] like Figure 2 As shown, Figure 2 This is a flowchart of a model optimization method according to an embodiment of this application, the model optimization method including the following steps: S201, FP32 model preparation.

[0084] S202, perform offline quantitative sensitivity analysis.

[0085] S203, Construct a learnable structured pruning mechanism.

[0086] S204, combined pruning and quantitative perception training.

[0087] S205, determine whether the number of training rounds is the maximum number of training rounds. If yes, proceed to step S211; otherwise, proceed to step S206.

[0088] S206, determine whether the current round is greater than the preset threshold N0. If yes, proceed to step S207; otherwise, return to step S204.

[0089] S207, Channel grouping and group mask update.

[0090] S208, determine whether the result of taking the modulus of the current training round with respect to the preset interval N1 is equal to 0. If yes, proceed to step S209; otherwise, return to step S204.

[0091] S209, Online Sensitivity Assessment.

[0092] S210, quantize bit width and pruning rate adjustment, and return to execution step S204.

[0093] S211, Model Export and Deployment.

[0094] According to the model optimization method proposed in this application, the initial quantization bit width and initial pruning rate of each convolutional layer in the BEV perception model are obtained. The weights of the convolutional layers are quantized based on the initial quantization bit width. The pruning mask of the output channel is obtained based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. The model is then trained based on the loss function, quantized weights, and pruning mask to obtain the initial deployment model. The initial deployment model is then trained with online quantization sensitivity evaluation. When the evaluation conditions are met, the initial deployment model is used as the final deployment model. This solves the problems of limited domain adaptation and generalization capabilities, decreased detection accuracy, difficulty in efficient utilization of the NPU of edge chips such as Horizon Robotics, inability to respond to model evolution during training, and severe error accumulation. It achieves structured sparsity, dynamic adaptation, and end-to-end collaborative optimization.

[0095] Next, the model optimization apparatus proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0096] Figure 3 A block diagram of a model optimization apparatus according to an embodiment of this application.

[0097] like Figure 3 As shown, the model optimization device 10 includes: an acquisition module 100, a quantization module 200, a training module 300, and an optimization module 400.

[0098] The module 100 acquires the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model. The BEV perception model includes multiple convolutional layers, and each convolutional layer includes weights and multiple output channels. The quantization module 200 quantizes the weights of each convolutional layer based on the initial quantization bit width of each convolutional layer to obtain the quantized weights of each convolutional layer, and obtains the pruning mask of the output channel of the corresponding convolutional layer based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. Training module 300 trains the BEV perception model based on the loss function of the BEV perception model, the quantized weights of each convolutional layer, and the pruning mask of the output channel until the preset training termination condition is met, so as to obtain the initial deployment model. The optimization module 400 performs online quantitative sensitivity evaluation training on the initial deployment model to obtain the evaluation results. When the evaluation results meet the preset evaluation conditions, the initial deployment model is used as the final deployment model.

[0099] According to one embodiment of this application, the quantization module 200 is specifically used for: Determine the importance score vector for each output channel of each convolutional layer; Based on the importance score vector of each output channel of each convolutional layer, the multiple output channels of each convolutional layer are sorted in descending order to obtain the ranking result of each convolutional layer. Based on the sorting results of each convolutional layer, the output channels of the corresponding convolutional layer are divided into groups to obtain multiple groups of channels for each convolutional layer. Calculate the average importance score vector for each group channel of each convolutional layer; The pruning mask for the output channel of each convolutional layer is obtained based on the initial pruning rate of each convolutional layer and the average importance score vector of the group channels.

[0100] According to one embodiment of this application, the quantization module 200 is specifically used for: Based on the importance score vector of the output channel of each convolutional layer, calculate the average importance score vector of the group channels of the corresponding convolutional layer; Based on the average importance score vector of the group channels of each convolutional layer, target groups whose average importance score vector is greater than or equal to the initial pruning rate of the corresponding convolutional layer are selected to obtain the target groups of each convolutional layer. Determine the group mask of the target group for each convolutional layer, and obtain the pruning mask for each output channel of each convolutional layer based on the group mask of the target group for each convolutional layer.

[0101] According to one embodiment of this application, the quantization module 200 is specifically used for: Based on the initial quantization bit width of each convolutional layer, pseudo-quantization nodes are inserted into the corresponding convolutional layer weights and activation values ​​during the forward pass, and quantization-dequantization is performed to generate the quantized weights of each convolutional layer.

[0102] According to one embodiment of this application, the optimization module 400 is specifically used for: Based on a preset sliding window, an inference is performed on the initial deployment model once according to a preset verification period and a preset verification set, resulting in multiple dequantized feature maps of each convolutional layer within the preset sliding window. The feature similarity of each convolutional layer within the preset sliding window is calculated based on the multiple dequantized feature maps of each convolutional layer within the preset sliding window. The sensitivity of each convolutional layer within the preset sliding window is obtained based on the multiple feature similarities of each convolutional layer within the preset sliding window. The average sensitivity of each convolutional layer is calculated based on the multiple sensitivities of each convolutional layer. The sensitivity is used to represent the degree of influence of the feature perturbation caused by the quantization operation of the corresponding convolutional layer on the final perception task accuracy. The higher the sensitivity, the more the convolutional layer needs to retain a high bit width, and vice versa. Based on the threshold buffer range of the average sensitivity of each convolutional layer, the target quantization bit width of each convolutional layer is determined, and the model is trained according to the target quantization bit width of each convolutional layer until the maximum number of training iterations is reached, resulting in the final deployment model.

[0103] According to one embodiment of this application, the loss function of the BEV perception model includes a prediction loss term and at least one regularization term, wherein the at least one regularization term includes a pruning regularization term, a quantization perception regularization term, and a spatial gradient regularization term, wherein... The pruning regularization term is configured to impose a sparse penalty on each output channel vector of each convolutional layer to drive the score of unimportant channels to monotonically decrease to zero during training. The quantization-aware regularization term is configured to establish a differentiable pass-through estimation difference measure between floating-point weights and quantized integer values; The spatial gradient regularization term is configured to apply a smoothing penalty to the first-order gradient magnitudes of the BEV feature map in the horizontal and vertical directions to reduce spatial discontinuities caused by quantization noise.

[0104] According to one embodiment of this application, the acquisition module 100 is specifically used for: Calculate the quantization sensitivity of each convolutional layer in the BEV perception model; The initial quantization bit width and initial pruning rate of each convolutional layer are obtained by initializing the quantization sensitivity of each convolutional layer.

[0105] According to one embodiment of this application, the acquisition module 100 is specifically used for: Based on the preset validation set, the BEV perception model is fully inferred, and the original task loss of each convolutional layer is obtained based on the inference results. The target convolutional layer is determined from the BEV perception model, and INT8 quantization is performed on the target convolutional layer while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model. The new BEV perception model is then subjected to full inference using a pre-set validation set, and the quantization task loss of the target convolutional layer is obtained based on the inference results. Determine a new convolutional layer from the remaining convolutional layers of the BEV perception model, use the new convolutional layer as the target convolutional layer, and re-execute the step of performing INT8 quantization on the target convolutional layer while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model, until the quantization task loss of each convolutional layer is obtained. The quantization sensitivity increment of each convolutional layer is calculated based on the quantization task loss and the original task loss of each convolutional layer, and then the quantization sensitivity increment of each convolutional layer is normalized to obtain the quantization sensitivity of each convolutional layer.

[0106] According to the model optimization apparatus proposed in this application, the initial quantization bit width and initial pruning rate of each convolutional layer in the BEV perception model are obtained. The weights of the convolutional layers are quantized based on the initial quantization bit width. The pruning mask of the output channel is obtained based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. The model is trained based on the loss function, quantized weights, and pruning mask to obtain an initial deployment model. The initial deployment model is then trained with online quantization sensitivity evaluation. When the evaluation conditions are met, the initial deployment model is used as the final deployment model. This solves the problems of limited domain adaptation and generalization capabilities, decreased detection accuracy, difficulty in efficient utilization of the NPU of edge chips such as Horizon Robotics, inability to respond to model evolution during training, and severe error accumulation. It achieves structured sparsity, dynamic adaptation, and end-to-end collaborative optimization.

[0107] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0108] When the processor 402 executes the program, it implements the model optimization method provided in the above embodiments.

[0109] Furthermore, electronic devices also include: Communication interface 403 is used for communication between memory 401 and processor 402.

[0110] The memory 401 is used to store computer programs that can run on the processor 402.

[0111] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0112] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0113] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0114] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0115] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model optimization method described above.

[0116] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model optimization method embodiments.

[0117] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0119] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A model optimization method, characterized in that, Includes the following steps: Obtain the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model, wherein the BEV perception model includes multiple convolutional layers, and each convolutional layer includes weights and multiple output channels; Based on the initial quantization bit width of each convolutional layer, the weights of the corresponding convolutional layer are quantized to obtain the quantized weights of each convolutional layer. The pruning mask of the output channel of the corresponding convolutional layer is obtained according to the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels. Based on the loss function of the BEV perception model, the quantized weights of each convolutional layer, and the pruning mask of the output channel, the BEV perception model is trained until the preset training termination condition is met to obtain the initial deployment model. The initial deployment model is trained using an online quantitative sensitivity assessment to obtain the assessment results. When the assessment results meet the preset assessment conditions, the initial deployment model is used as the final deployment model.

2. The method according to claim 1, characterized in that, The step of obtaining the pruning mask for the output channel of the corresponding convolutional layer based on the initial pruning rate of each convolutional layer and the importance score vector of multiple output channels includes: Determine the importance score vector for each output channel of each of the convolutional layers; Based on the importance score vector of each output channel of each convolutional layer, the multiple output channels of each convolutional layer are sorted in descending order to obtain the sorting result of each convolutional layer; Based on the sorting result of each convolutional layer, the output channels of the corresponding convolutional layer are grouped to obtain multiple group channels for each convolutional layer. Calculate the average importance score vector for each group of channels in each of the convolutional layers; The pruning mask for the output channel of each convolutional layer is obtained based on the initial pruning rate of each convolutional layer and the average importance score vector of the group channels.

3. The method according to claim 2, characterized in that, The step of obtaining the pruning mask for the output channel of each convolutional layer based on the initial pruning rate of each convolutional layer and the average importance score vector of the group channels includes: Based on the importance score vector of the output channel of each convolutional layer, calculate the average importance score vector of the group channels of the corresponding convolutional layer; Based on the average importance score vector of the group channels of each convolutional layer, target groups whose average importance score vector is greater than or equal to the initial pruning rate of the corresponding convolutional layer are selected to obtain the target groups of each convolutional layer; Determine the group mask of the target group for each convolutional layer, and obtain the pruning mask for each output channel of each convolutional layer based on the group mask of the target group for each convolutional layer.

4. The method according to claim 1, characterized in that, The step of quantizing the weights of each convolutional layer based on the initial quantization bit width of each convolutional layer to obtain the quantized weights of each convolutional layer includes: Based on the initial quantization bit width of each convolutional layer, pseudo-quantization nodes are inserted into the corresponding convolutional layer weights and activation values ​​during the forward pass, and quantization-dequantization is performed to generate the quantized weights of each convolutional layer.

5. The method according to claim 1, characterized in that, The step of training the initial deployment model with online quantitative sensitivity evaluation to obtain evaluation results, and using the initial deployment model as the final deployment model when the evaluation results meet preset evaluation conditions, includes: Based on a preset sliding window, an inference is performed on the initial deployment model once according to a preset verification period and a preset verification set to obtain multiple dequantized feature maps of each convolutional layer within the preset sliding window; The feature similarity of each convolutional layer within the preset sliding window is calculated based on the multiple dequantized feature maps of each convolutional layer within the preset sliding window. Multiple sensitivities of each convolutional layer within the preset sliding window are obtained based on these multiple feature similarities. The average sensitivity of each convolutional layer is then calculated based on these multiple sensitivities. The sensitivity represents the degree to which the feature perturbation caused by the quantization operation of the corresponding convolutional layer affects the final accuracy of the perception task. Higher sensitivity requires a higher bit width for the convolutional layer, while lower sensitivity allows for a lower bit width. Based on the threshold buffer where the average sensitivity of each convolutional layer is located, the target quantization bit width of each convolutional layer is determined, and the model is trained according to the target quantization bit width of each convolutional layer until the maximum number of training iterations is reached, thus obtaining the final deployment model.

6. The method according to claim 1, characterized in that, The loss function of the BEV perception model includes a prediction loss term and at least one regularization term, wherein the at least one regularization term includes a pruning regularization term, a quantization perception regularization term, and a spatial gradient regularization term, wherein... The pruning regularization term is configured to apply a sparsity penalty to each output channel vector of each of the convolutional layers to drive the score of unimportant channels to monotonically decrease to zero during training. The quantization-aware regularization term is configured to establish a differentiable pass-through estimation difference measure between the floating-point weights and the quantized integer values; The spatial gradient regularization term is configured to apply a smoothing penalty to the first-order gradient magnitudes of the BEV feature map in the horizontal and vertical directions to reduce spatial discontinuities caused by quantization noise.

7. The method according to claim 1, characterized in that, The process of obtaining the initial quantization bit width and initial pruning rate of each convolutional layer in the top-view BEV perception model includes: Calculate the quantization sensitivity of each convolutional layer in the BEV perception model; The initial quantization bit width and initial pruning rate of each convolutional layer are obtained by initializing the quantization sensitivity of each convolutional layer.

8. The method according to claim 7, characterized in that, The calculation of the quantization sensitivity of each convolutional layer in the BEV perception model includes: Based on a preset validation set, the BEV perception model is subjected to complete inference, and the original task loss of each convolutional layer is obtained based on the inference results. The target convolutional layer is determined from the BEV perception model, and the target convolutional layer is quantized using INT8 while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model. The new BEV perception model is then subjected to full inference using the preset validation set, and the quantization task loss of the target convolutional layer is obtained based on the inference result. A new convolutional layer is determined from the remaining convolutional layers of the BEV perception model, and the new convolutional layer is used as the target convolutional layer. The step of performing INT8 quantization on the target convolutional layer is re-executed while keeping the parameters and structure of the remaining convolutional layers unchanged to obtain a new BEV perception model, until the quantization task loss of each convolutional layer is obtained. The quantization sensitivity increment of each convolutional layer is calculated based on the quantization task loss and the original task loss of each convolutional layer, and the quantization sensitivity increment of each convolutional layer is normalized to obtain the quantization sensitivity of each convolutional layer.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the model optimization method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the model optimization method as described in any one of claims 1-8.

Images