Hardware perception based neural network mixed precision quantization method, device and equipment

Abstract

This invention provides a hardware-aware neural network hybrid precision quantization method, apparatus, and device, relating to the fields of artificial intelligence and edge computing. The method includes: acquiring the hardware architecture parameters of the target edge device where the neural network is deployed; based on the hardware architecture parameters, analyzing the memory, inference latency, and power consumption constraints of the target edge device, and combining this with the quantization bit width range supported by the target edge device, constructing constraints; performing multi-dimensional sensitivity analysis on the quantization error of each layer of the neural network, and constructing a precision loss function for the neural network by considering the quantization error introduced when different bit widths are used in each layer; constructing an optimization problem based on the constraints, with the goal of minimizing the precision loss function, to quantize the hybrid precision allocation of the bit width of each layer of the neural network; and solving the optimization problem to obtain the optimal quantization bit width corresponding to each layer of the neural network. This invention ensures a balance between computational efficiency, memory usage, and precision.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and edge computing technology, and in particular to a hardware-aware neural network hybrid precision quantization method, apparatus and device. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale language models are being used more and more widely in fields such as computer vision, natural language processing, and the Internet of Things. However, the high computational complexity and massive memory requirements of these models create a sharp contradiction with the limited computing power, memory capacity, and power consumption budget of edge devices, becoming a core bottleneck for the deployment of large models on the edge, necessitating model quantization.

[0003] Static quantization, a type of model quantization technique, has become a common solution for edge computing scenarios due to its low deployment cost and the fact that it eliminates the need for model retraining. Its core principle is to achieve model compression and accelerated inference by reducing the precision of model parameters and computational data.

[0004] However, static quantization uses a globally uniform quantization bit width. This quantization method either uses a low bit width to pursue the ultimate compression rate and inference speed, resulting in model accuracy loss that exceeds the acceptable range, or uses a high bit width to ensure core accuracy, sacrificing compression effect and running efficiency. Ultimately, it is difficult to meet the comprehensive deployment requirements of edge devices for large models with high accuracy, high efficiency and low memory usage. Summary of the Invention

[0005] This invention provides a hardware-aware neural network hybrid precision quantization method, apparatus, and device to address the imbalance between computational efficiency, memory usage, and precision in traditional neural network quantization methods.

[0006] In a first aspect, embodiments of the present invention provide a hardware-aware hybrid precision quantization method for neural networks, comprising: acquiring hardware architecture parameters of the target edge device where the neural network is deployed; based on the hardware architecture parameters, constructing constraints by analyzing the memory, inference latency, and power consumption constraints of the target edge device, and combining the quantization bit width range supported by the target edge device; performing multidimensional sensitivity analysis on the quantization error of each layer of the neural network, and constructing a precision loss function of the neural network by combining the quantization error introduced when different bit widths are used in each layer of the neural network; constructing an optimization problem for the hybrid precision allocation of the bit width of each layer of the neural network based on the constraints, with the goal of minimizing the precision loss function; and solving the optimization problem to obtain the optimal quantization bit width corresponding to each layer of the neural network.

[0007] In one possible implementation, based on hardware architecture parameters, constraints are constructed through analysis of the target device's memory, inference latency, and power consumption, combined with the quantization bit width range supported by the target device. These constraints include: determining the target device's memory threshold, inference latency threshold, and power consumption threshold based on hardware architecture parameters; calculating the memory usage of each layer based on data transfer volume and candidate bit widths; constructing memory constraints based on the sum of memory usage across all layers and the memory threshold; calculating the inference latency of each layer based on the number of multiply-accumulate operations, hardware-specific bit width computation throughput, data transfer volume, and memory bandwidth; constructing inference latency constraints based on the sum of inference latency across all layers and the inference latency threshold; determining the unit power consumption and data transfer power consumption of each bit width in each layer based on hardware architecture parameters and micro-benchmark tests; calculating the power consumption of each layer based on the unit power consumption and data transfer power consumption of each bit width in each layer; constructing power consumption constraints based on the sum of power consumption of all layers and the power consumption threshold; and constructing bit width constraints based on the low-precision bit width range compatible with the target device.

[0008] In one possible implementation, a multidimensional sensitivity analysis is performed on the quantization error of each layer of the neural network. Combining this with the quantization error introduced by different bit widths used in each layer, a precision loss function for the neural network is constructed. This includes: for each layer, calculating the Hessian sensitivity of that layer based on the trace of the second derivative matrix of the loss function with respect to the weights of that layer. The Hessian sensitivity characterizes the degree of second-order influence of weight parameter perturbations on the loss function. Calculating the distribution characteristics of the activation values ​​of that layer based on the maximum value and the median absolute value of the activation values ​​of that layer. This distribution characteristic characterizes the quantization difficulty of that layer. Calculating the gradient sensitivity based on the standard deviation and the mean absolute value of the weight gradients of that layer. This gradient sensitivity characterizes the sensitivity of that layer to quantization errors. Calculating the comprehensive sensitivity score of that layer based on the Hessian sensitivity, the distribution characteristics of the activation values ​​of that layer, and the gradient sensitivity. Finally, constructing the precision loss function of the neural network based on the quantization error introduced by each layer when using a preset candidate bit width, combined with the comprehensive sensitivity score of each layer.

[0009] In one possible implementation, before calculating the Hessian sensitivity of each layer of the neural network based on the trace of the second derivative matrix of the loss function with respect to the weights of that layer, the method further includes: calculating the KL divergence based on the activation value distribution of the current layer and the activation value distribution of the calibration data; if the KL divergence is greater than a preset threshold, calculating the updated adaptation factor based on the KL divergence and the initial adaptation factor; and updating the model weights of that layer based on the updated adaptation factor and the floating-point weights of that layer.

[0010] In one possible implementation, before calculating the KL divergence based on the activation value distribution of the current layer and the activation value distribution of the calibration data, the method further includes: determining the truncation range of the activation values ​​of the current layer based on a preset quantile; truncating the activation values ​​based on the truncation range to obtain truncated activation values; and determining the activation value distribution of the current layer based on the truncated activation values.

[0011] In one possible implementation, after calculating the model weights of the layer based on the updated adaptation factor and the floating-point weights of the layer, the method further includes: calculating the cumulative error introduced by all previous layers; calculating the calculation result of the current layer based on the input activation value of the current layer and the model weights of the current layer; and calculating the output activation value of the current layer based on the calculation result of the current layer and the cumulative error, which is then used as the input activation value of the next layer.

[0012] In one possible implementation, the optimization problem is solved to obtain the optimal quantization bit width for each layer of the neural network. This includes: calculating the equivalent precision loss for each layer when selecting each candidate bit width based on the hardware performance model, the comprehensive sensitivity score of each layer, and the quantization error, memory usage, inference latency, and energy consumption corresponding to different candidate bit widths; based on the equivalent precision loss, with the goal of minimizing the overall precision loss of the neural network, and under the premise of satisfying the constraints of memory, inference latency, and energy consumption of the target edge device, a dynamic programming algorithm is used to perform state transitions and backtracking to obtain the optimal quantization bit width for each layer.

[0013] In one possible implementation, the candidate bit width includes 4 bits, 8 bits, and 16 bits.

[0014] Secondly, embodiments of the present invention provide a hardware-aware neural network hybrid precision quantization device, comprising: a communication module for acquiring hardware architecture parameters of a target edge device on which the neural network is deployed; a processing module for constructing constraints based on the hardware architecture parameters, through memory, inference latency, and power consumption constraints of the target edge device, combined with the quantization bit width range supported by the target edge device; performing multidimensional sensitivity analysis on the quantization error of each layer of the neural network, and constructing a precision loss function of the neural network by combining the quantization error introduced when different bit widths are used in each layer of the neural network; constructing an optimization problem for the hybrid precision allocation of the bit width of each layer of the neural network based on the constraints, with the goal of minimizing the precision loss function; and solving the optimization problem to obtain the optimal quantization bit width corresponding to each layer of the neural network.

[0015] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0016] This invention provides a hardware-aware neural network hybrid precision quantization method, apparatus, and device. First, constraints are constructed using the hardware architecture parameters of the edge device to clarify the hardware constraint boundaries of the quantization process. This ensures that subsequent bit width allocation does not exceed the device's memory, latency, and power consumption budget, while fully utilizing the quantization bit width types supported by the device, achieving a precise match between the quantization scheme and hardware resources. Then, through multi-dimensional sensitivity analysis of each layer, the differences in sensitivity to quantization errors among layers are accurately distinguished. This ensures that the precision loss function reflects the greater importance of highly sensitive layers to quantization errors, avoiding poor local precision due to neglecting inter-layer differences. Finally, within the hardware constraints, precision loss is minimized by allocating quantization bit widths appropriate to the sensitivity of each layer, ultimately achieving a balance between high precision, high computational efficiency, low memory usage, and low power consumption. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the implementation of the hardware-aware neural network hybrid precision quantization method provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the hardware-aware neural network hybrid precision quantization device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] See Figure 1 The flowchart illustrating the implementation of the hardware-aware neural network hybrid precision quantization method provided in this embodiment of the invention is described in detail below: Step 101: Obtain the hardware architecture parameters of the target edge device for deploying the neural network.

[0020] In some embodiments, a neural network is an artificial intelligence model composed of multiple layers of computing units, such as convolutional layers, fully connected layers, and attention layers, used to achieve functions such as feature extraction and inference prediction. This invention refers to a large-scale language model or a multimodal model.

[0021] In some embodiments, the target edge device refers to a resource-constrained device that deploys a neural network, such as a smartphone, IoT device, edge AI box, embedded system, etc., as opposed to a cloud server.

[0022] In some embodiments, hardware architecture parameters are core hardware characteristic data of the edge device.

[0023] For example, hardware architecture parameters include processor type and instruction set architecture, supported quantization operation types and throughput, memory hierarchy, memory bandwidth, real-time power budget, thermal design power, etc.

[0024] In some embodiments, the hardware architecture parameter collection is specifically as follows: First, the hardware architecture parameters of the edge device are examined, including: processor type and instruction set architecture, supported quantization operation types and their throughput, memory hierarchy, real-time power budget, and thermal design power. Then, a hardware performance model is constructed to establish the mapping relationship between quantization bit width and actual inference latency and power consumption.

[0025] in , , , The hardware correlation coefficient is obtained through micro-benchmark testing. This indicates the number of multiply-accumulate operations that need to be performed at this layer. This indicates the computing throughput of the hardware at a specific bandwidth. This indicates the amount of data that this layer needs to transmit. This indicates memory bandwidth. This indicates the energy consumption per unit when performing this bit-width operation. This indicates the energy consumption for transmitting data.

[0026] Step 102: Based on hardware architecture parameters, construct constraints by analyzing the memory, inference latency, and power consumption constraints of the target edge device, combined with the quantization bit width range supported by the target edge device.

[0027] In some embodiments, the quantization bit width range is the set of quantization data bits supported by the edge device hardware. The smaller the bit width, the lower the data storage and computation cost, but it may introduce a larger quantization error.

[0028] In some embodiments, constraints are hard limitations that ensure the quantized model can run stably on the edge device and are the core basis for subsequent bit width allocation.

[0029] As one possible implementation, step 102 can be specifically implemented as steps 1021-1029.

[0030] 1021: Based on hardware architecture parameters, determine the memory threshold, inference latency threshold, and power consumption threshold of the target edge device.

[0031] In some embodiments, the memory threshold is the maximum available memory capacity that the target edge device can stably support, and is the upper limit of the total memory usage of the quantized model.

[0032] In some embodiments, the inference latency threshold is the maximum acceptable time for the edge device to perform neural network inference, ensuring that the model meets real-time requirements.

[0033] In some embodiments, the power consumption threshold is the maximum power consumption limit that the edge device can withstand during inference, adapting to the device's battery life or heat dissipation capabilities.

[0034] In this embodiment, the threshold is determined based on hardware parameters to ensure that the memory of the quantized model does not exceed the limit or the latency is not too high, thus solving the problem of existing technologies ignoring the actual capabilities of the hardware.

[0035] 1022: Calculate the memory usage of each layer based on the data transmission volume and candidate bit width of each layer.

[0036] In some embodiments, the data transmission volume of each layer is the total amount of data that a single layer of the neural network needs to read and write during inference.

[0037] For example, the data transmission volume of each layer includes weight data, activation value data, etc. Specifically, weight data may include the weight matrix of the convolutional layer and the weight matrix of the fully connected layer. Activation value data may include the input feature map and output feature map of the fully connected layer.

[0038] In some embodiments, the candidate bit width is a low-precision quantization bit width compatible with the edge device hardware, and is a key parameter for calculating memory usage.

[0039] In some embodiments, the memory footprint of each layer is the memory storage space required after a single layer is quantized using a certain candidate bit width.

[0040] In this embodiment, by breaking down the memory usage of each layer, high memory consumption layers can be accurately identified, laying the foundation for subsequent mixed precision allocation, such as the reasonable selection of bit width for high memory layers, and avoiding the problem of overall memory exceeding the limit without being able to locate the issue.

[0041] 1023: Construct memory constraints based on the sum of memory usage across all layers and memory thresholds.

[0042] In some embodiments, the memory constraint is a limitation that the total memory usage of all layers of the quantized neural network does not exceed the maximum available memory of the edge device.

[0043] In this embodiment, by constraining the total memory usage, it is ensured that the model will not crash due to insufficient memory after quantization, while providing clear restrictions on subsequent bit width allocation.

[0044] 1024: Calculate the inference latency of each layer based on the number of multiply-accumulate operations, hardware-specific bandwidth computing throughput, data transfer volume, and memory bandwidth.

[0045] In some embodiments, the number of multiply-accumulate operations is a core metric for measuring the computational complexity of a layer. The number of multiply-accumulate operations equals the total number of multiplication-addition operations required for a single layer of the neural network to complete one inference operation.

[0046] In some embodiments, the hardware-specific bit width computing throughput is the number of multiply-accumulate operations that the target end device can complete per unit time under a certain quantization bit width, reflecting the computing efficiency of the hardware under that bit width.

[0047] In some embodiments, memory bandwidth is the data transfer rate between the device's memory and the processor, which determines the speed of data reading and writing.

[0048] In some embodiments, the inference latency of each layer is the time it takes for a single layer to complete inference after quantization, reflecting the computation and data transmission time of a single layer.

[0049] In this embodiment, by combining the throughput and memory bandwidth of the hardware-specific bit width, the latency calculation is made to fit the actual hardware performance, avoiding the disconnect between general latency estimation and actual hardware performance, and providing a basis for subsequent hardware-aware bit width allocation.

[0050] 1025: Construct inference delay constraints based on the sum of inference delays across all layers and the inference delay threshold.

[0051] In some embodiments, the inference latency constraint is a limitation that the total time for the neural network to complete one full inference computation does not exceed the acceptable upper limit of the edge device.

[0052] In this embodiment, by constraining the total latency, the model is ensured to meet the real-time response requirements of the edge device, while guiding subsequent bit width allocation. For example, in latency-sensitive scenarios, bit widths that are hardware-efficiently supported are prioritized for computationally intensive layers.

[0053] 1026: Based on hardware architecture parameters and combined with micro-benchmark tests, determine the power consumption of each bit width in each layer for computation and data transmission.

[0054] In some embodiments, the power consumption per unit of computation is the power consumption consumed by the hardware to perform a multiplication and addition operation on a unit number of units at a certain quantization bit width, reflecting the energy efficiency of the computation operation.

[0055] In some embodiments, data transmission power consumption refers to the power consumption consumed by the hardware to complete the transmission of a unit amount of data under a certain quantization bit width, reflecting the energy efficiency of data reading and writing.

[0056] In this embodiment, accurate energy consumption parameters are obtained through micro-benchmark testing, avoiding the problem of general energy consumption estimation not matching the actual hardware, so that subsequent energy consumption calculations are more in line with the actual battery life of the device.

[0057] 1027: Calculate the energy consumption of each layer based on the energy consumption of each bit width operation unit and the energy consumption of data transmission.

[0058] In some embodiments, the energy consumption of each layer is the total power consumption consumed during inference after a single layer is quantized with a certain candidate bit width.

[0059] In this embodiment, the energy consumption of each layer is broken down, which can accurately identify high-energy-consuming layers, such as computationally intensive convolutional layers, and provide a basis for subsequent mixing precision allocation. For example, high-energy-consuming layers can be selected with more energy-efficient bit widths to avoid exceeding the overall energy consumption limit without being able to optimize.

[0060] 1028: Construct energy consumption constraints based on the total energy consumption of all layers and the energy consumption threshold.

[0061] In some embodiments, the energy consumption constraint is a limitation that the total power consumption consumed during neural network inference does not exceed the power budget of the edge device.

[0062] For example, the power consumption budget of an edge device may include battery life and heat dissipation capabilities.

[0063] In this embodiment, by constraining the total energy consumption, the model is ensured to run for a long time on battery-powered edge devices or to work stably in embedded systems with limited heat dissipation. At the same time, it forms a triple guarantee with memory constraints and latency constraints to avoid other resource overruns caused by a single constraint.

[0064] 1029: Construct bit width constraints based on the low-precision bit width range compatible with the target end device.

[0065] In some embodiments, the set of quantization bit widths supported by the target end device hardware for low-precision bit width range excludes bit widths that are hardware incompatible.

[0066] In this embodiment, unsupported bit widths are excluded to solve the problem of incompatibility between the quantization bit width and the hardware, which leads to inference failure. At the same time, the range of candidate bit widths is limited to reduce the search space for subsequent optimization problems and improve the solution efficiency.

[0067] In this embodiment, memory constraints prevent excessive memory usage after model quantization, which could lead to device lag or crashes. Inference latency constraints meet the real-time requirements of the edge device. Energy consumption constraints adapt to the battery life or heat dissipation capabilities of the edge device. Bit width constraints ensure that the quantization scheme matches the hardware support type, avoiding invalid quantization operations.

[0068] Step 103: Perform multidimensional sensitivity analysis on the quantization error of each layer of the neural network, and construct the accuracy loss function of the neural network by combining the quantization error introduced when different bit widths are used in each layer of the neural network.

[0069] In some embodiments, multidimensional sensitivity analysis assesses the tolerance of each layer of a neural network to quantization errors from multiple dimensions.

[0070] For example, multidimensional sensitivity analysis includes Hessian sensitivity, activation value dynamic range, and gradient sensitivity.

[0071] As one possible implementation, step 103 can be specifically implemented as steps 1031-1035.

[0072] 1031: For each layer of the neural network, the Hessian sensitivity of that layer is calculated based on the trace of the second derivative matrix of the loss function with respect to the weights of that layer. The Hessian sensitivity characterizes the degree of second-order influence of the perturbation of the weight parameters on the loss function.

[0073] In some embodiments, the trace of the second derivative matrix is ​​obtained by taking the second partial derivative of the loss function with respect to all weights in a certain layer, forming a Hessian matrix, and then summing the diagonal elements of the matrix to quantify the impact of small changes in weights on the loss function.

[0074] In some embodiments, Hessian sensitivity is a sensitivity index obtained by the trace of the Hessian matrix. The larger the value, the more a small perturbation of the weights of the layer will cause a large change in the loss function. That is, the layer is more sensitive to quantization error, and accuracy should be prioritized during quantization.

[0075] In some embodiments, different layers of a neural network have vastly different tolerances to quantization errors. Therefore, a multi-dimensional sensitivity assessment is performed on each layer of the neural network, calculating Hessian sensitivity and the loss function. Regarding weight The second derivative of the hessian matrix, i.e., the trace of the Hessian matrix, characterizes the second-order effect of parameter perturbation on the loss function:

[0076] In this embodiment, the impact of weight perturbation on the loss is quantified mathematically to avoid accuracy errors caused by mistakenly assigning low bit widths to important layers.

[0077] 1032: Based on the maximum absolute value of the activation values ​​of this layer and the median absolute value of the activation values ​​of this layer, the distribution characteristics of the activation values ​​of this layer are calculated. The distribution characteristics represent the quantization difficulty of this layer.

[0078] In some embodiments, activation values ​​are intermediate data output after the input data of a certain layer of a neural network is processed by weight calculation and activation function, such as feature map data after the ReLU function, which is a data type that needs to be processed synchronously during the quantization process.

[0079] In some embodiments, the distribution characteristics of activation values ​​are calculated by dividing the maximum absolute value of activation values ​​by the logarithm of the median absolute value of activation values. This reflects the data distribution range of activation values. The larger the value, the more dispersed the distribution of activation values, indicating the existence of extreme values. This makes it easier to generate errors during quantization and increases the difficulty of quantization.

[0080] In some embodiments, the dynamic range of activation values ​​is analyzed, and the distribution characteristics of activation values ​​in each layer are statistically analyzed using a calibration dataset.

[0081] in, For the first The larger the dynamic range of the layer activation value, the higher the difficulty of quantization.

[0082] In this embodiment, the quantization distribution characteristics provide a basis for the bit width allocation of this type of layer, such as allocating a high bit width or optimizing the quantization range, which solves the problem that the existing technology ignores the impact of activation value distribution on the quantization effect.

[0083] 1033: Based on the standard deviation of the weight gradient and the mean of the absolute values ​​of the gradients of this layer, the gradient sensitivity is calculated. The gradient sensitivity characterizes the degree to which this layer is sensitive to quantization errors.

[0084] In some embodiments, gradient sensitivity is calculated by dividing the standard deviation of the weight gradient by the mean of the absolute values ​​of the gradients. The larger the value, the greater the fluctuation of the weight gradient of that layer, and the more significant the impact of the weight precision loss caused by quantization on the model performance, that is, the lower the tolerance to quantization error.

[0085] In some embodiments, gradient sensitivity is evaluated by calculating the statistical properties of the weight gradient, as follows:

[0086] in, This represents the weight gradient; the larger the coefficient of variation, the more sensitive the layer is to quantization error.

[0087] In this embodiment, the sensitive layer is verified by gradient characteristics, forming a dual verification with Hessian sensitivity, which improves the accuracy of sensitive layer identification and avoids the one-sidedness of single-dimensional evaluation.

[0088] 1034: Based on Hessian sensitivity, the distribution characteristics of activation values ​​in this layer, and gradient sensitivity, the overall sensitivity score of this layer is calculated.

[0089] In some embodiments, the overall sensitivity score is the final index obtained by weighted fusion of Hessian sensitivity, activation value distribution characteristics, and gradient sensitivity. It comprehensively reflects the overall sensitivity of the layer to quantization error. The higher the score, the more bit width the layer needs to be allocated during quantization to reduce accuracy loss; the lower the score, the more bit width can be allocated to save resources.

[0090] In some embodiments, the final comprehensive sensitivity score is calculated:

[0091] in, , , The weighting coefficients were determined through small-scale experiments.

[0092] In this embodiment, by weighted fusion of three sensitivity indicators, the limitations of a single indicator are avoided, making the importance ranking of layers more comprehensive and accurate, and providing a core basis for the subsequent differentiated allocation of high bit width for sensitive layers and low bit width for non-sensitive layers.

[0093] 1035: Based on the quantization error introduced by each layer when using a preset candidate bit width, and combined with the comprehensive sensitivity score of each layer, the accuracy loss function of the neural network is constructed.

[0094] In some embodiments, quantization error is the error caused by the loss of data precision when converting floating-point weights or activation values ​​of a neural network into low-precision quantized values ​​such as INT4 / INT8.

[0095] In some embodiments, the preset candidate bit width is a set of quantization bit widths supported by the end device.

[0096] In some embodiments, quantization error is the precision loss caused by converting floating-point data to low-precision data when a layer is quantized with a specific candidate bit width, such as the numerical deviation of weights / activation values, which can be obtained through offline calibration experiments.

[0097] In this embodiment, the precision loss function clarifies the impact of different bit width combinations on the overall precision, making the subsequent optimization problem—that is, minimizing the precision loss under constraints of memory, latency, and energy consumption—operable. This solves the problem of the lack of clear target guidance in the existing technology of mixed precision allocation, avoids blindly allocating bit width, and ensures that the precision loss of the model after quantization is minimized.

[0098] In this embodiment, multidimensional sensitivity analysis accurately identifies the tolerance of each layer to quantization errors. Sensitive layer errors have a greater impact, while non-sensitive layer errors have a smaller impact. The accuracy loss function correlates layer sensitivity with quantization errors, making subsequent optimization objectives more closely match the actual accuracy requirements of the model, avoiding sacrificing sensitive layer accuracy for compression ratio or wasting non-sensitive layer resources for accuracy.

[0099] Step 104: Construct an optimization problem for the mixed precision allocation of each layer of the quantized neural network, with the objective of minimizing the precision loss function and based on constraints.

[0100] In some embodiments, hybrid precision allocation is distinct from global uniform bit width quantization. Different quantization bit widths are allocated to different layers of the neural network, such as 16 bits for sensitive layers and 4 bits for non-sensitive layers, to achieve a balance between precision and resource consumption.

[0101] In some embodiments, the optimization problem is to find a mathematical model of the bit width allocation scheme of each layer that minimizes the overall accuracy loss, while satisfying constraints on memory, latency, and energy consumption.

[0102] In some embodiments, a precision allocation optimization problem is constructed, with the goal of minimizing precision loss while satisfying hardware constraints, i.e., minimizing the sum of weighted quantization errors across all layers:

[0103]

[0104] in, This indicates the importance of the layer. This indicates the error introduced when using this bit width for quantization.

[0105] In this embodiment, the constraints and accuracy loss function of the preceding steps are integrated to form a dialectical model that minimizes accuracy loss and satisfies resource constraints, providing a clear mathematical framework for the subsequent solution of the optimal bit width.

[0106] Step 105: Solve the optimization problem to obtain the optimal quantization bit width for each layer of the neural network.

[0107] As one possible implementation, step 105 can be specifically implemented as steps 1051-1052.

[0108] 1051: Based on the hardware performance model, the comprehensive sensitivity scores of each layer, and the quantization error, memory usage, inference latency and power consumption corresponding to different candidate bit widths, calculate the equivalent precision loss when selecting each candidate bit width for each layer.

[0109] In some embodiments, the hardware performance model is a mathematical model constructed through hardware feature acquisition and micro-benchmark testing, which establishes a mapping relationship between quantization bit width and inference latency and energy consumption, and is the core basis for computing layer resource consumption.

[0110] In some embodiments, the overall sensitivity score of each layer is an indicator obtained by weighted fusion of Hessian sensitivity, activation value distribution characteristics, and gradient sensitivity. The higher the score, the more sensitive the layer is to quantization error, and the more important it is to ensure accuracy.

[0111] In some embodiments, memory usage is the storage space required for data such as weights and activation values ​​when a certain layer uses a specific candidate bit width.

[0112] In some embodiments, the comprehensive cost index of equivalent precision loss, fusion layer importance, quantization inherent error and hardware adaptability is calculated as comprehensive sensitivity score × quantization error × hardware matching factor. The smaller the value, the better the adaptability of the bit width to the layer.

[0113] In some embodiments, the candidate bit width includes 4 bits, 8 bits, and 16 bits.

[0114] In this embodiment, a comprehensive sensitivity score is integrated to ensure that the quantization error of the sensitivity layer is given priority consideration. Combined with a hardware matching factor, the bit width efficiently supported by the hardware is prioritized, addressing the problem of existing technologies neglecting hardware heterogeneity. The coordinated evaluation of quantization error, memory, latency, and power consumption avoids single-dimensional optimization, laying the foundation for subsequent multi-constraint solutions.

[0115] 1052: Based on equivalent precision loss, with the goal of minimizing the overall precision loss of the neural network, under the premise of meeting the constraints of memory, inference latency and energy consumption of the target edge device, a dynamic programming algorithm is used to solve the state transition and backtracking problem to obtain the optimal quantization bit width for each layer.

[0116] In some embodiments, the overall accuracy loss is the sum of the equivalent accuracy losses of all layers in the neural network, and is the objective function of the optimization problem.

[0117] In some embodiments, the optimal quantization bit width is the quantization bit width that minimizes the overall precision loss for each layer of the neural network while satisfying all constraints, such as 4 bits, 8 bits, or 16 bits.

[0118] In some embodiments, dynamic programming is a deterministic algorithm that solves combinatorial optimization problems through state definition-initialization-transition-backtracking. Its core is to efficiently find the global optimal solution by utilizing the characteristics of optimal substructure and overlapping subproblems.

[0119] In some embodiments, state transition is the core iterative process of dynamic programming, which traverses all reachable states at each level, i.e., the cumulative resource consumption and minimum loss of the previous level, tries all candidate bit widths, calculates the cumulative resources and total loss of the new state, and updates and retains the optimal state.

[0120] In some embodiments, backtracking is derived from the optimal termination state of the last layer, and the optimal quantization bit width of each layer is determined sequentially through the recorded transition paths.

[0121] In some implementations, the optimal quantization bit width is the candidate bit width for each layer that minimizes the overall accuracy loss of the neural network while satisfying all constraints.

[0122] In this embodiment, compared to the greedy algorithm which yields a locally optimal solution, the particle swarm optimization algorithm yields a near-optimal solution, ensuring that the final bit-width combination is the globally optimal solution, maximizing the balance between accuracy and resource consumption. By discretizing the state and reusing sub-problems, the exponential complexity of the enumeration method is avoided, ensuring that the edge device can quickly complete the solution and meet the requirements for rapid deployment after training. During the state transition process, memory, latency, and energy consumption constraints are directly verified, and only feasible states are retained. The backtracking results naturally satisfy all constraints, preventing the model from failing to run due to excessive resources after quantization.

[0123] In this embodiment, the optimal bit width allocation scheme ensures that the sensitive layer uses a high bit width to reduce accuracy loss, the non-sensitive layer uses a low bit width to save resources, and the bit width that is efficiently supported by the hardware is given priority to improve inference performance, ultimately achieving efficient deployment of large models with lightweight and high accuracy on the edge device.

[0124] This invention provides a hardware-aware neural network hybrid precision quantization method, apparatus, and device. First, constraints are constructed using the hardware architecture parameters of the edge device to clarify the hardware constraint boundaries of the quantization process. This ensures that subsequent bit width allocation does not exceed the device's memory, latency, and power consumption budget, while fully utilizing the quantization bit width types supported by the device, achieving a precise match between the quantization scheme and hardware resources. Then, through multi-dimensional sensitivity analysis of each layer, the differences in sensitivity to quantization errors among layers are accurately distinguished. This ensures that the precision loss function reflects the greater importance of highly sensitive layers to quantization errors, avoiding poor local precision due to neglecting inter-layer differences. Finally, within the hardware constraints, precision loss is minimized by allocating quantization bit widths appropriate to the sensitivity of each layer, ultimately achieving a balance between high precision, high computational efficiency, low memory usage, and low power consumption.

[0125] As one possible implementation, steps 201-203 are included before step 1031.

[0126] 201: Based on the activation value distribution of the current layer and combined with the activation value distribution of the calibration data, the KL divergence is calculated.

[0127] In some embodiments, the activation value distribution of the current layer is the probability distribution of the activation values ​​generated by the current layer of the neural network after processing the actual input data during inference, reflecting the feature distribution of the real-time input data.

[0128] In some embodiments, the activation value distribution of the calibration data is obtained offline using the calibration dataset, which is representative data that matches the model training / application scenario. The current layer activation value probability distribution is then used as a benchmark to measure whether the real-time data distribution has shifted.

[0129] In some embodiments, KL divergence, or relative entropy, is an indicator that measures the degree of difference between two probability distributions. The larger the value, the greater the deviation between the real-time input data distribution and the baseline distribution, and direct quantization can easily introduce a large number of errors.

[0130] In some embodiments, the quantization parameters are dynamically adjusted based on the actual input data. First, the input data distribution is detected, and the KL divergence between the current batch input and the calibration data is calculated.

[0131] in, KL divergence is used to measure the difference between two probability distributions. To calibrate the distribution, the activation value distribution was statistically analyzed using calibration data during the offline phase. This represents the current input distribution, indicating the distribution of activation values ​​for the current batch during runtime.

[0132] In this embodiment, when the real-time data is out-of-distribution data, the KL divergence will increase significantly. Timely capture of this difference can avoid the risk of a sharp drop in quantization accuracy in advance.

[0133] 202: If the KL divergence is greater than the preset threshold, calculate the updated adaptation factor based on the KL divergence and the initial adaptation factor.

[0134] In some embodiments, the preset threshold is a critical value for distribution difference determined by offline experiments, used to determine whether the real-time data distribution has shifted to the point where the quantization parameters need to be adjusted.

[0135] In some embodiments, the initial adaptation factor is a quantization scaling factor determined based on calibration data in the offline phase. It is used to map floating-point activation values / weights to a low-precision quantization range and is the core parameter of the initial quantization.

[0136] In some embodiments, the updated adaptation factor is a dynamically adjusted quantization scaling factor that combines the degree of distribution difference, i.e., KL divergence, when the KL divergence exceeds a threshold, to ensure that the quantization range fits the real-time data distribution.

[0137] In some embodiments, when the KL divergence exceeds a preset threshold θ, the quantization scaling factor, i.e., the adaptation factor, is updated:

[0138] in, Here is the adjustment coefficient. This represents a new quantification factor.

[0139] In this embodiment, when the real-time data distribution shifts, the updated adaptation factor will expand or shrink the quantization range to prevent the activation value / weight from being truncated due to the distribution shift exceeding the original quantization range, thus solving the core problem that fixed quantization parameters cannot adapt to dynamic input.

[0140] 203: Based on the updated adaptation factor and combined with the floating-point weights of this layer, the model weights of this layer are updated to obtain the model weights of this layer.

[0141] In some embodiments, floating-point weights are the unquantized raw weight data of the current layer, typically 32-bit / 64-bit floating-point data, and are the core learnable parameters for feature transformation in the model.

[0142] In some embodiments, the model weights are weight data obtained by quantizing the floating-point weights with the updated adaptation factor, which retains the core features of the weights and adapts to the real-time data distribution, reducing quantization errors.

[0143] In some embodiments, the process by which the adaptation factor affects the weights is as follows:

[0144] in, and These represent the weights before and after quantization, respectively.

[0145] In this embodiment, the updated model weights are more closely aligned with the characteristics of the current input data. Subsequent Hessian sensitivity calculations can more accurately reflect the quantization sensitivity of this layer to the current data, avoiding sensitivity assessment bias caused by mismatch between weights and data distribution. Weights that are adapted to the data distribution in advance introduce less error during subsequent quantization, indirectly improving the overall quantization accuracy of the model and echoing the beneficial effect of the patent's strong accuracy retention capability.

[0146] As one possible implementation, steps 301-303 are included before step 201.

[0147] 301: Determine the cutoff range of the activation values ​​of the current layer based on the preset quantile.

[0148] In some embodiments, the preset quantile is a pre-set statistical quantile threshold, such as the 99.9 quantile, used to filter a reasonable range of data among the active values ​​and remove extreme outliers. A quantile is a critical value used to divide data into different proportions after sorting it by size.

[0149] In some embodiments, the truncation range is the numerical boundary of the activation value determined based on the 99.9 quantile. The portion of the absolute value of the activation value that exceeds this range will be truncated, and only the valid data within the range will be retained.

[0150] In some embodiments, outliers are removed using the following truncation method:

[0151] The 99.9 quantile was used to determine the cutoff range to reduce the impact of outliers.

[0152] In this embodiment, a preset 99.9 percentile is used as the preset percentile to ensure that 99.9% of the effective activation values ​​are retained, and only 0.1% of extreme outliers are removed. This not only preserves the characteristics of the effective data, but also accurately identifies the outliers that need to be removed. This solves the problem that the cutoff range of the prior art does not have a unified standard, which can easily lead to the loss of effective data or the retention of outliers.

[0153] 302: The activation value is truncated based on the truncation range to obtain the truncated activation value.

[0154] In some embodiments, truncation is an operation that limits the value of the activation value according to a set truncation range. If the absolute value of the activation value is greater than the upper limit of the truncation range, it is adjusted to the upper limit of the truncation range; if it is less than or equal to the upper limit, the original value is retained. The core purpose is to remove the interference of extreme outliers on the data distribution.

[0155] In some embodiments, the truncated activation values ​​are activation value data obtained after truncation operation, which has eliminated extreme outliers, resulting in a more concentrated and stable data distribution and avoiding the problem of a surge in quantization error caused by outliers.

[0156] In this embodiment, extreme outliers, such as extremely large activation values ​​caused by abnormal inputs, can cause the activation value distribution to exhibit false dispersion characteristics, and direct quantization will introduce a large number of errors. By truncating, activation values ​​that exceed the reasonable range are restricted to within the truncation threshold, making the activation value data distribution more concentrated and more in line with the true effective characteristics, thus clearing away data interference for subsequent distribution analysis and quantization parameter adjustment.

[0157] 303: Determine the activation value distribution of the current layer based on the truncated activation values.

[0158] In some embodiments, the activation value distribution is the probability distribution of truncated activation values ​​within a numerical range, such as values ​​concentrated between 0 and 5, or exhibiting a normal or uniform distribution, etc., and is a core indicator for measuring data characteristics.

[0159] In some embodiments, the activation value distribution of the current layer differs from the original activation value distribution. This distribution only reflects the characteristics of valid data, excludes the interference of extreme outliers, and serves as an accurate benchmark for subsequent calculation of KL divergence and determination of whether the data distribution has shifted.

[0160] In this embodiment, the truncated activation value distribution eliminates the interference of extreme outliers and can truly reflect the effective feature distribution of the current layer input data. Based on this, the KL divergence can be calculated to accurately determine the distribution difference between the real-time input data and the calibration data, ensuring that subsequent adaptation factor updates and weight adjustments are triggered only when adaptation is truly needed, thereby improving the accuracy and robustness of dynamic adjustment of quantization parameters.

[0161] As one possible implementation, steps 401-403 are included after step 203.

[0162] 401: Statistically sum the cumulative error introduced by all preceding layers.

[0163] In some embodiments, all preceding layers refer to all layers in the neural network before the current layer. For example, when processing the 5th layer, the preceding layers are layers 1-4. Quantization operations on these layers will generate errors that will propagate gradually.

[0164] In some embodiments, the cumulative error is the total error value after the sum of the quantization errors of all preceding layers. It is the sum of errors propagating across layers. If it is not compensated, the accuracy of subsequent layers will continue to deteriorate.

[0165] In this embodiment, the error of a single-layer quantization may be small, but the superposition of multiple layers will significantly distort the data characteristics. This step makes the compensation operation targeted rather than blindly corrected by statistically accumulating the error.

[0166] 402: The calculation result of the current layer is obtained based on the input activation value of the current layer and the model weight of the current layer.

[0167] In some embodiments, the model weights of this layer are updated model weights, quantized weights adapted to the real-time data distribution, and are the core parameters for feature transformation of the current layer.

[0168] In some embodiments, the current layer calculation result is the original result of the current layer after completing the basic operation by input activation value × model weight + bias, without error compensation, and is the basis for subsequent error compensation.

[0169] In some embodiments, since quantization errors accumulate and propagate, a quantization error accumulation compensation mechanism is further introduced. This mechanism records the error introduced by each quantization layer and compensates for it in subsequent layers through bias terms. The accumulated error is then added to the calculation of the next layer.

[0170] in, This represents the quantitative calculation results of the standard. For compensation coefficient, This represents the accumulated error from the preceding layers.

[0171] In this embodiment, the calculation result is the ideal operation output of the current layer that is not affected by the accumulated error. As a reference for subsequent compensation, only by clarifying the original result can the amount of compensation for the accumulated error be accurately superimposed, so as to ensure that the compensated data corrects the error without deviating from the true characteristics.

[0172] 403: Based on the calculation results of the current layer and combined with the cumulative error, the output activation value of the current layer is calculated and used as the input activation value of the next layer.

[0173] In some embodiments, the output activation value is the final output data after superimposing the current layer calculation result with the previous cumulative error compensation, which has corrected the quantization error propagating across layers.

[0174] In some embodiments, the input activation value of the next layer is the output activation value of the current layer, which is directly used as the input data of the next layer. This ensures that the subsequent layers receive accurate data after error compensation, breaking the chain of error accumulation and amplification.

[0175] In this embodiment, the negative impact of quantization errors in previous layers is directly offset by the calculation results of the current layer and the accumulated error compensation term, solving the core problem that the error accumulates larger and larger in later layers. The compensated output activation value is used as the input of the next layer, ensuring that the calculation of subsequent layers is based on accurate data, so that the effects of sensitivity analysis and mixed precision allocation are not canceled out by the error, indirectly ensuring the accuracy and stability of the entire quantization scheme; this compensation mechanism does not require retraining the model, but is calculated in real time only during inference, reducing computational costs.

[0176] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0177] The above embodiments are in Figure 1 Based on the method shown, each step will be discussed in detail. To facilitate understanding of the complete execution process, the overall method flow will be discussed below with reference to an embodiment.

[0178] Deploying visual-language multimodal models on edge AI boxes Application scenario: Deploying visual-language multimodal models to edge computing devices.

[0179] Implementation steps: Step 1: Hardware Architecture Parameter Acquisition The device was detected to have a CPU and GPU, with a throughput of 40 TOPS, 8GB of available memory, and a GPU memory bandwidth of 68 GB / s.

[0180] Step 2: Multimodal Sensitivity Analysis Sensitivity evaluations were performed on image encoders and text encoders respectively. It was found that the early convolutional layers and patch embedding of the image encoder are highly sensitive to quantization, while the attention layer of the text encoder is generally more sensitive than the FFN layer.

[0181] Step 3: Cross-modal accuracy allocation The specific quantization precision allocation is as follows: Image Patch Embedding: FP16, Image ViT first 6 layers: INT8 weights + activation, Image ViT last 18 layers: INT8 weights + INT4 activation, Text Encoder: all INT8, Cross-modal projection layer: FP16.

[0182] Step 4: Modality-aware dynamic quantization For image input, the quantization strategy is dynamically adjusted based on image complexity: simple images use INT4 quantization, and complex images use INT8 quantization. For text input, the quantization parameters of the embedding layer are adjusted based on word rarity.

[0183] Step 5: Heterogeneous Execution Optimization The image and text encoders are executed in parallel, using DLA (Deep Learning Accelerator) to perform INT8 convolutions, while the CPU is responsible for post-processing and result aggregation.

[0184] Step 6: Multimodal alignment quality monitoring Monitor the numerical stability of the image-text similarity matrix and trigger quantization backoff when abnormal similarity scores are detected.

[0185] Experimental results: The model size was compressed from 1.7GB to 550MB, and the inference latency for image-text pairs decreased from 145ms to 52ms. Classification accuracy on ImageNet decreased from 76.2% to 75.6%. Power consumption decreased from 10W to 4.2W.

[0186] Compared with existing technologies, this invention has the following significant advantages: First, improved compression efficiency. Through a hybrid precision strategy, model size compression is achieved while maintaining minimal precision loss, significantly improving the compression ratio compared to uniform quantization methods. Second, optimized inference performance. Hardware-aware precision allocation further reduces inference latency and energy consumption, fully utilizing the computing power of edge devices. Third, strong precision preservation capability. Multi-dimensional sensitivity analysis and dynamic quantization parameter adjustment mechanisms ensure minimal precision loss of the quantized model across various tasks, significantly outperforming traditional methods. Fourth, good hardware adaptability. It automatically adapts to the hardware characteristics of different edge devices, eliminating the need for manual tuning for each device. Fifth, strong runtime robustness. The dynamic adjustment mechanism enables the system to adapt to changes in input data distribution and device state fluctuations, significantly improving stability in practical applications.

[0187] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0188] Figure 2 A schematic diagram of the hardware-aware neural network hybrid precision quantization device provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 2 As shown, the hardware-aware neural network hybrid precision quantization device 2 includes: Communication module 21 is used to obtain the hardware architecture parameters of the target edge device where the neural network is deployed; Processing module 22 is used to construct constraints based on hardware architecture parameters, memory, inference latency, and power consumption constraints of the target edge device, combined with the quantization bit width range supported by the target edge device; perform multi-dimensional sensitivity analysis on the quantization error of each layer of the neural network, and construct the precision loss function of the neural network by combining the quantization error introduced when different bit widths are used in each layer of the neural network; construct an optimization problem of mixed precision allocation of the bit width of each layer of the quantization neural network with the objective of minimizing the precision loss function, based on the constraints; solve the optimization problem to obtain the optimal quantization bit width corresponding to each layer of the neural network.

[0189] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Figure 3 As shown, the electronic device 3 of this embodiment includes a processor 30 and a memory 31. The memory 31 stores a computer program 32. When the processor 30 executes the computer program 32, it implements the steps in the various method embodiments described above. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the various device embodiments described above.

[0190] For example, computer program 32 may be divided into one or more modules / units, which are stored in memory 31 and executed by processor 30 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 32 in electronic device 3.

[0191] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 3 may also include input / output devices, network access devices, buses, etc.

[0192] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0193] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0194] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A hardware-aware neural network hybrid precision quantization method, characterized in that, include: Obtain the hardware architecture parameters of the target edge device for deploying the neural network; Based on the hardware architecture parameters, constraints are constructed by analyzing the memory, inference latency, and power consumption of the target edge device, combined with the quantization bit width range supported by the target edge device. A multidimensional sensitivity analysis of the quantization error of each layer of the neural network is performed, and the accuracy loss function of the neural network is constructed by combining the quantization error introduced when different bit widths are used in each layer of the neural network. Construct an optimization problem for the mixed precision allocation of each layer of a quantized neural network, with the objective of minimizing the precision loss function and based on the constraints. The optimization problem is solved to obtain the optimal quantization bit width for each layer of the neural network.

2. The hardware-aware based neural network mixed precision quantization method according to claim 1, wherein, Based on the hardware architecture parameters, and through the analysis of memory, inference latency, and power consumption constraints of the target edge device, combined with the quantization bit width range supported by the target edge device, constraints are constructed, including: Based on the hardware architecture parameters, determine the memory threshold, inference latency threshold, and power consumption threshold of the target edge device; Calculate the memory usage of each layer based on the data transmission volume and candidate bit width of each layer; Memory constraints are constructed based on the sum of memory usage across all layers and the aforementioned memory threshold. The inference latency of each layer is calculated based on the number of multiply-accumulate operations at each layer, the hardware-specific bandwidth computing throughput, the amount of data transfer, and the memory bandwidth. Based on the sum of inference delays across all layers and the inference delay threshold, an inference delay constraint is constructed. Based on hardware architecture parameters and combined with micro-benchmark tests, the energy consumption of computing unit and data transmission of each bit width in each layer is determined. Calculate the energy consumption of each layer based on the energy consumption of each bit width operation unit and the energy consumption of data transmission. Energy consumption constraints are constructed based on the sum of energy consumption of all layers and the energy consumption threshold. A bit width constraint is constructed based on the low-precision bit width range compatible with the target end device.

3. The hardware-aware based neural network mixed precision quantization method according to claim 1, wherein, The method involves performing multidimensional sensitivity analysis on the quantization errors of each layer of the neural network, and combining this with the quantization errors introduced by different bit widths used in each layer of the neural network to construct the accuracy loss function of the neural network, including: For each layer of the neural network, the Hessian sensitivity of that layer is calculated based on the trace of the second derivative matrix of the loss function with respect to the weights of that layer. The Hessian sensitivity characterizes the degree of second-order influence of the perturbation of the weight parameters on the loss function. Based on the maximum absolute value of the activation values ​​of the layer and the median absolute value of the activation values ​​of the layer, the distribution characteristics of the activation values ​​of the layer are calculated, and the distribution characteristics characterize the quantization difficulty of the layer. The gradient sensitivity is calculated based on the standard deviation of the weight gradient and the mean of the absolute values ​​of the gradients of this layer. The gradient sensitivity characterizes the degree to which this layer is sensitive to quantization errors. Based on the Hessian sensitivity, the distribution characteristics of the activation values ​​of the layer, and the gradient sensitivity, the overall sensitivity score of the layer is calculated. Based on the quantization error introduced by each layer when using a preset candidate bit width, and combined with the comprehensive sensitivity score of each layer, the accuracy loss function of the neural network is constructed.

4. The hardware-aware neural network hybrid precision quantization method according to claim 3, characterized in that, Before calculating the Hessian sensitivity of each layer of the neural network based on the trace of the second derivative matrix of the loss function with respect to the weights of that layer, the following steps are also included: Based on the activation value distribution of the current layer and the activation value distribution of the calibration data, the KL divergence is calculated. If the KL divergence is greater than a preset threshold, the updated adaptation factor is calculated based on the KL divergence and the initial adaptation factor. Based on the updated adaptation factor, combined with the floating-point weights of the layer, the model weights of the layer are updated to obtain the updated model weights.

5. The hardware-aware neural network hybrid precision quantization method according to claim 4, characterized in that, Before calculating the KL divergence based on the activation value distribution of the current layer and the activation value distribution of the calibration data, the following steps are also included: Based on the preset quantiles, determine the cutoff range of the activation values ​​for the current layer; The activation value is truncated based on the truncation range to obtain the truncated activation value; Based on the truncated activation values, the activation value distribution of the current layer is determined.

6. The hardware-aware neural network hybrid precision quantization method according to claim 4, characterized in that, After calculating the model weights of the layer based on the updated adaptation factor and the floating-point weights of that layer, the process further includes: The cumulative error introduced by all preceding layers is statistically analyzed. The calculation result of the current layer is obtained based on the input activation value of the current layer and the model weight of the current layer; Based on the calculation results of the current layer and the accumulated error, the output activation value of the current layer is calculated and used as the input activation value of the next layer.

7. The hardware-aware based neural network mixed precision quantization method according to claim 1, wherein, Solving the optimization problem to obtain the optimal quantization bit width for each layer of the neural network includes: Based on the hardware performance model, the comprehensive sensitivity scores of each layer, and the quantization error, memory usage, inference latency and energy consumption corresponding to different candidate bit widths, the equivalent precision loss when selecting each candidate bit width for each layer is calculated. Based on the equivalent accuracy loss, with the goal of minimizing the overall accuracy loss of the neural network, and under the premise of satisfying the constraints of memory, inference latency and energy consumption of the target edge device, a dynamic programming algorithm is used to perform state transition and backtracking solution to obtain the optimal quantization bit width for each layer.

8. The hardware-aware neural network hybrid precision quantization method according to claim 7, characterized in that, The candidate bit widths include 4 bits, 8 bits, and 16 bits.

9. A hardware-aware neural network mixed precision quantization apparatus, comprising: include: The communication module is used to obtain the hardware architecture parameters of the target edge device where the neural network is deployed; The processing module is configured to: construct constraints based on the hardware architecture parameters, including memory, inference latency, and power consumption constraints of the target edge device, and the quantization bit width range supported by the target edge device; perform multi-dimensional sensitivity analysis on the quantization error of each layer of the neural network, and construct a precision loss function for the neural network by considering the quantization error introduced when different bit widths are used in each layer; construct an optimization problem for the mixed precision allocation of the bit width of each layer of the neural network based on the constraints, with the goal of minimizing the precision loss function; and solve the optimization problem to obtain the optimal quantization bit width corresponding to each layer of the neural network.

10. An electronic device, comprising: It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 8.

Images