An FP16 accuracy compensation system and method for Ascend NPU

Abstract

The application discloses an FP16 precision compensation system and method for Ascend NPU, and belongs to the field of artificial intelligence. The method comprises the following steps: dividing an input tensor to be calculated and a weight tensor into a plurality of blocks; determining a dynamic scaling factor based on statistical characteristics of the blocks, and performing scaling processing on the blocks; performing matrix multiplication or convolution operation on the scaled blocks on a matrix calculation unit of the Ascend NPU to obtain basic output blocks; extracting error characteristics of the basic output blocks on a vector calculation unit, and comparing the error characteristics with a preset threshold to obtain a trigger flag; if the trigger flag meets a compensation condition, selecting the first K components from the basic output blocks and performing residual compensation operation, fusing the compensation result with the basic output blocks to obtain compensation output blocks; and taking the compensation output blocks as the current layer output and entering the next layer of inference calculation. The application can effectively reduce the FP16 precision operation error on the Ascend NPU end side.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence inference acceleration and numerical calculation optimization technology, specifically involving an FP16 accuracy compensation method and system for Ascend NPU. Background Technology

[0002] With the widespread application of deep neural networks in edge computing and edge scenarios, model inference typically needs to be performed in real-time or near real-time under limited computing power, storage, and power consumption. To reduce inference computational overhead, low-precision data formats such as FP16 are commonly used in engineering implementations to improve throughput and reduce storage bandwidth usage. However, the FP16 format has limited exponent and mantissa bit widths, which can easily lead to rounding error accumulation, numerical underflow / overflow, and insufficient dynamic range in high-density multiplication and addition operations such as matrix multiplication and convolution. This can result in increased model output bias, manifesting as decreased inference accuracy or unstable results.

[0003] In the prior art, the following methods are usually adopted to address the problem of decreased accuracy in low-precision inference: (1) Using a hybrid precision strategy, higher precision calculations are used in some operators or some layers to alleviate error accumulation; (2) The model is adapted to low-precision representation during the training phase through offline quantization calibration, quantization-aware training, etc.; (3) Error compensation or correction mechanisms are introduced during the inference process to correct the low-precision calculation error.

[0004] The above methods can improve low-precision inference accuracy under certain conditions, but they still have at least the following shortcomings or limitations in edge deployment scenarios:

[0005] Firstly, in edge scenarios, mixed-precision solutions often require the introduction of higher-precision operators or additional data format conversion and data handling, which can easily increase computation and memory access overhead, thereby affecting the stable throughput under real-time and power consumption constraints.

[0006] Secondly, offline quantization calibration and quantization-aware training methods usually rely on additional training / calibration processes and data preparation. Furthermore, when the input distribution or operating conditions change during the inference phase, certain accuracy fluctuations may still occur. In addition, these methods are usually based on static strategies at the model level or level, making it difficult to handle a small number of "instantaneous high-risk" error components as needed during the inference process.

[0007] Third, in the existing inference-side error compensation or correction mechanisms, the compensation calculation in some implementations still mainly involves the full processing of the entire layer or block output, which leads to the additional computing power and memory access overhead increasing with the output scale, which is not conducive to edge deployment and latency jitter control.

[0008] Therefore, there is an urgent need for a precision compensation scheme for the Ascend NPU edge FP16 inference scenario, which can at least alleviate the error risk caused by the limited dynamic range and the accumulation of rounding errors in FP16 inference. Without significantly increasing the inference computation and memory access overhead, the scheme can correct the error on demand through dynamic scaling and triggered Top-K selective residual compensation, thereby improving the stability and accuracy of the inference results. Summary of the Invention

[0009] The purpose of this invention is to provide an FP16 accuracy compensation method and system for Ascend NPU, so as to solve the problems of decreased inference accuracy and unstable results caused by limited dynamic range and accumulated rounding errors in existing edge-side FP16 inference.

[0010] This invention introduces a dynamic scaling and triggered Top-K residual compensation mechanism to selectively correct low-precision calculation errors while controlling additional computation and memory access overhead, thereby improving edge inference accuracy and operational stability.

[0011] Technical solution of the present invention

[0012] A method for FP16 accuracy compensation for Ascend NPU, the method comprising the following steps:

[0013] Step 1: Obtain the input tensor and weight tensor to be performed inference calculation, and divide the input tensor and weight tensor into a set of input blocks and a set of weight blocks respectively according to a predetermined block division strategy, and establish the correspondence between the input blocks and the weight blocks to form a block pair set.

[0014] Step 2: For the input block corresponding to the block index in the current layer and its corresponding weight block, calculate the statistical characteristics of the input block and / or its corresponding weight block, and determine the dynamic scaling factor based on the statistical characteristics;

[0015] Step 3: Perform dynamic scaling processing on the input block and / or its corresponding weight block according to the dynamic scaling factor to obtain the input block and weight block after dynamic scaling processing;

[0016] Step 4: On the matrix calculation unit of the Ascend NPU, perform matrix multiplication or convolution operations on the input block and weight block after the dynamic scaling process to obtain the basic output block; wherein, the dynamic scaling process causes one of the input block and / or the weight block to be scaled, while the other remains unchanged or is scaled inversely.

[0017] Step 5: On the vector computing unit of Ascend NPU, error features are extracted based on the basic output block, and optionally combined with auxiliary features extracted from the input block corresponding to the basic output block to determine the error features; the error features are compared with a preset threshold to obtain a trigger flag;

[0018] Step 6: If the trigger flag meets the preset compensation condition, then select K candidate components from the basic output block as Top-K components according to the preset importance index, and perform residual compensation operation on the Top-K components based on the compensation coefficient parameters determined in the offline calibration stage to obtain the compensated components.

[0019] Step 7: Merge the compensation component with the basic output block to obtain the compensation output block; otherwise, directly use the basic output block as the compensation output block.

[0020] Step 8: Write the compensated output block into the corresponding position of the current layer output tensor; if there are still unprocessed block indices in the current layer, update the block index and return to step 2 to process the next block pair; when all block pairs in the current layer have been processed, obtain the current layer output tensor, and use the current layer output tensor as the input for the next layer inference calculation, return to step 1 to process the next layer, until the inference calculation is completed, and the program ends.

[0021] Furthermore, the statistical features in step 2 include at least one of the following: the maximum absolute value of the input block, the mean and variance of the input block, and the exponential range representation of the input block; the maximum absolute value of the weight block, the mean and variance of the weight block, and the exponential range representation of the weight block.

[0022] Further, the method for determining the dynamic scaling factor in step 2 includes: calculating a candidate value set of scaling factors based on the statistical characteristics, and selecting from the candidate value set a scaling factor that causes the numerical range of the input block and / or its corresponding weight block after dynamic scaling to fall within a preset FP16 safety range as the dynamic scaling factor; wherein, the candidate value set of scaling factors includes at least a scaling factor calculated by any of the following statistical measures: maximum absolute value, root mean square value, absolute value quantile, and a combined pruning threshold of the maximum absolute value and the quantile; and / or, the candidate value set of scaling factors may also include candidate scaling factors determined by FP16 exponential range constraints, compensation trigger thresholds, and Top-K compensation budget parameters; wherein, the statistical measures Characterizes the magnitude scale of the input block and / or its corresponding weighted block, and sets the target upper bound or target scale corresponding to the preset FP16 safety range as... Then the candidate scaling factor satisfies ,in To prevent division by zero constants, the candidate scaling factor can be further quantized into an integer power of two form. Where n is limited to a preset power range Integers within, where For the least power, The maximum power is defined as the maximum power. The preset FP16 safety range is determined by the pre-calibrated exponent interval and the requirement to retain the effective mantissa.

[0023] The statistic A can also be the fusion effective amplitude A. fuse The fused effective amplitude is composed of the maximum absolute value m, the root mean square r, and the absolute value quantile threshold q. PK The result of fusion is: A fuse =(1-β)·A tail +β·r, where A tail =min(m,ρ eff ·q PK ) ; p K =1-K / N; ρ eff =clip(ρ0+ρ1· (1-p K )+ρ2· I(E prev ≤θ),ρ min ,ρ max ), β∈[0,1] are the fusion weights; E prev I(·) is the feedback quantity of the error characteristics or triggering result of the previous block, and I(·) is the indicator function. The candidate scaling factor can be set according to A=A fuse Substitute the values ​​into the formula and quantize them into an integer power of two.

[0024] Furthermore, the third step of performing dynamic scaling on the input block and / or its corresponding weight block includes: performing scaling on the input block by multiplying by the dynamic scaling factor, and performing scaling on the weight block by multiplying by the reciprocal of the dynamic scaling factor; or, performing scaling on the weight block by multiplying by the dynamic scaling factor, and performing scaling on the input block by multiplying by the reciprocal of the dynamic scaling factor; to ensure that the desired output scale of the matrix multiplication or convolution operation remains unchanged.

[0025] Furthermore, the fifth step of extracting error features includes: calculating energy representation, dispersion representation, and / or saturation ratio representation for the basic output block on the vector calculation unit, and generating the error features based on a preset combination rule; wherein, the saturation ratio representation is used to represent the proportion of elements in the basic output block whose absolute value is greater than the saturation threshold; the preset combination rule includes at least normalizing and then weighting the representations, or performing vector concatenation and then linear mapping to obtain the error feature value.

[0026] Furthermore, the selection criteria for the Top-K components in step 6 include at least one of the following or a combination thereof: (1) the absolute value of each component in the basic output block; (2) the compensation priority score of each component in the basic output block; (3) the saturation risk of each component in the basic output block; wherein, the compensation priority score includes at least: a score obtained by weighted fusion of the absolute value of the component and the error characteristics of the corresponding channel; the saturation risk includes at least: the ratio of the absolute value of the component to the saturation threshold, or a function value representing the distance between the component exponent bit and the exponent boundary that can be represented by FP16; the index set and / or mask set corresponding to the Top-K components of the Top-K selection output are used to indicate the calculation position of the residual compensation operation.

[0027] Furthermore, the residual compensation operation in step 6 includes: calculating the compensation component based on the current value of the Top-K component and the error characteristics according to a preset piecewise function; wherein the preset piecewise function includes at least two different linear mapping relationships, and the coefficients of each linear mapping relationship are determined by the compensation coefficient parameters determined in the offline calibration stage; the compensation coefficient parameters determined in the offline calibration stage include at least a piecewise threshold τ, a first linear mapping coefficient a1 and a first bias term b1, a second linear mapping coefficient a2 and a second bias term b2; the compensation coefficient parameters are obtained through offline calibration, which includes at least: obtaining the base output and reference output based on calibration data and calculating the residuals, and performing piecewise fitting on the residuals to determine the piecewise threshold and the coefficients and bias terms of each linear mapping; the compensation coefficient parameters determined in the offline calibration stage are stored layer by layer in the parameter storage device; the calculation of the compensation component is implemented on the vector calculation unit through element-wise multiplication and addition operations.

[0028] The preset piecewise function can be explicitly expressed as: for any component in the Top-K index set, the residual estimate d is obtained. i (For example, from the reference value / high-precision estimate Y) ref,i With basic output Y 0,i The difference is obtained, or the residual mapping model obtained from offline calibration is used to represent Y. 0,i (Obtained by estimating the error characteristic E), and then calculating the compensation component Δ. i =sign(d i ) · g(|d i |), where g(|d|) = a1· |d| + b1 (|d| ≤ τ), g(|d|) = a2· |d| + b2 (|d| > τ); the compensation output is Y. i =Y 0,I +Δ i For the components that are not in the Top-K, let Δ i =0.

[0029] Furthermore, steps 4 and 5 are executed in a collaborative pipelined manner on the Ascend NPU, including: while the matrix computation unit performs matrix multiplication or convolution operations on the current input block and weight block, the vector computation unit performs error feature extraction and residual compensation operations on the basic output block of the previous block; and the input block, weight block, and basic output block satisfy a predetermined alignment constraint so that the block dimension of the matrix multiplication or convolution operation matches the data layout of the vector compensation operation.

[0030] An FP16 precision compensation system for Ascend NPU, the system includes an inference execution device and a parameter storage device; wherein, the parameter storage device is used to store threshold parameters, Top-K filtering parameters, compensation coefficient parameters and scaling strategy parameters corresponding to each layer;

[0031] The reasoning execution device includes:

[0032] The block segmentation module is used to obtain the input tensor and weight tensor to be performed inference computation, and divide the input tensor and weight tensor into a set of input blocks and a set of weight blocks respectively according to a predetermined block segmentation strategy, and establish a correspondence between the input blocks and the weight blocks to form a block pair set.

[0033] The feature calculation module is used to calculate the statistical features of the input block and / or its corresponding weight block for any block index in any network layer.

[0034] A scaling factor determination module is used to determine a dynamic scaling factor based on the statistical characteristics;

[0035] The dynamic scaling module is used to perform dynamic scaling processing on the input block and / or its corresponding weight block according to the dynamic scaling factor, so as to obtain the input block and weight block after dynamic scaling processing.

[0036] The matrix calculation module is used to perform matrix multiplication or convolution operations on the dynamically scaled input blocks and weight blocks using the matrix calculation unit of the Ascend NPU to obtain the basic output blocks.

[0037] An error feature extraction module is used to extract error features based on the basic output block using the vector computing unit of the Ascend NPU, and optionally combine the auxiliary features extracted from the input block corresponding to the basic output block to determine the error features;

[0038] The trigger discrimination module is used to compare the error characteristics with a preset threshold to obtain a trigger flag;

[0039] The Top-K filtering module is used to filter K candidate components from the basic output block as Top-K components according to a preset importance index when the trigger flag meets the preset compensation conditions.

[0040] The residual compensation module is used to perform residual compensation calculations on the Top-K components based on the compensation coefficient parameters determined in the offline calibration stage to obtain the compensated components;

[0041] The fusion module is used to fuse the compensation component with the basic output block to obtain the compensation output block;

[0042] The output and scheduling module is used to write the compensated output blocks into the corresponding positions of the current layer output tensor, and to schedule each module to execute in a loop when there are unprocessed block indices in the current layer; when all block pairs in the current layer have been processed, the current layer output tensor is output and used as the input for the next layer inference calculation, until the inference calculation is completed.

[0043] The specific content involved in this invention and the meaning of the terms used are as follows:

[0044] To facilitate understanding and implementation of the technical solution of this invention, some terms involved in this invention are explained as follows; without ambiguity, the terms described in this specification have the following meanings:

[0045] (1) "Ascend NPU": refers to the Ascend series neural network processor or its equivalent implementation, used to perform deep learning inference computation, including at least a matrix computation unit for matrix multiplication and addition operations and a vector computation unit for vector element-wise AND and reduction operations.

[0046] (2) "Matrix computation unit": refers to the hardware computation unit in Ascend NPU used to perform matrix multiplication, convolution and other matrix multiplication and addition operations. It is used to perform matrix multiplication or convolution operations on input blocks and weight blocks and output basic output blocks.

[0047] (3) "Vector computing unit": refers to the hardware computing unit in Ascend NPU used to perform vector operations such as element-wise multiplication and addition, statistical calculation, and threshold comparison. It is used to extract error features, perform trigger discrimination, and perform residual compensation operations.

[0048] (4) "Input tensor / input block": refers to the input data of the current layer of the neural network, including but not limited to activation features, embedded features or intermediate features; the data format of the input tensor is FP16 or can be converted to FP16; the input block refers to the local data block obtained from the input tensor according to the block strategy, and the input block is one of the operation inputs of the matrix calculation unit.

[0049] (5) "Weight tensor / weight block": refers to the parameter data of the current layer of the neural network, including but not limited to convolutional kernel weights, fully connected weights or attention projection weights; the data format of the weight tensor is FP16 or can be converted to FP16; the weight block refers to the local data block obtained from the weight tensor according to the block strategy, and the weight block is one of the operation inputs of the matrix calculation unit.

[0050] (6) "Basic output block": refers to the output block obtained by the matrix calculation unit after performing matrix multiplication or convolution operation on the scaled input block and the scaled weight block. The basic output block is the calculation result before compensation.

[0051] (7) "Blocking strategy": refers to the rules for dividing the input tensor and / or weight tensor into several blocks. The rules include at least the block dimension, block size and block alignment constraint. The block alignment constraint is used to match the block dimension of the matrix calculation unit with the data layout of the vector calculation unit.

[0052] (8) "Statistical characteristics": refers to the statistical quantities used to characterize the numerical distribution of the input blocks and / or their corresponding weighted blocks, including at least the maximum absolute value, mean, variance and exponential range representation; wherein, the exponential range representation is used to characterize the exponential distribution range of the block data in FP16 representation; the exponential range representation can be determined by the exponential statistics obtained by taking the logarithm of the absolute value of the block data, or by the minimum and maximum values ​​of the exponential bits after converting the block data into FP16 representation.

[0053] (9) "Dynamic scaling factor": refers to the scaling coefficient determined according to statistical characteristics. Under the current block index, the dynamic scaling object satisfies any of the following modes: (a) only the input block is dynamically scaled, while the weight block remains unchanged; (b) only the weight block is dynamically scaled, while the input block remains unchanged; (c) both the input block and the weight block are dynamically scaled simultaneously; and the corresponding matrix multiplication / convolution calculation is performed in the mode to obtain the basic output block, so that the scaled value range falls within the preset FP16 safety range.

[0054] (10) "Dynamic scaling processing": refers to the processing of multiplicative scaling and / or reverse scaling of the input block and / or its corresponding weight block according to the dynamic scaling factor, so as to ensure that the expected output scale of matrix multiplication or convolution operation remains unchanged.

[0055] (11) "Preset FP16 safe range": refers to the stable range of FP16 values ​​obtained by pre-calibration, which is determined by the exponential interval and the requirement to retain the effective mantissa.

[0056] (12) "Error characteristics": refers to the characteristic quantity used to characterize the risk level of overflow, underflow or rounding error amplification of the basic output block under FP16 calculation conditions. The error characteristics are calculated by the vector calculation unit and used to trigger discrimination.

[0057] (13) "Preset threshold": refers to the threshold parameter used to compare error characteristics. The threshold parameter can be configured independently for each layer or determined according to the block strategy and scaling strategy.

[0058] (14) “Saturation threshold”: refers to the threshold used to calculate the saturation ratio, which is used to characterize the judgment criterion that the absolute value of the basic output block element reaches or exceeds the saturation risk range.

[0059] (15) "Trigger flag": refers to the judgment result obtained by comparing the error characteristics with the preset threshold, which is used to characterize whether the current basic output block meets the preset compensation conditions.

[0060] (16) "Preset compensation conditions": refers to the set of conditions used to determine whether to perform residual compensation operation, including at least the result of trigger flag satisfying threshold judgment.

[0061] (17) "Top-K components": refers to the top K components selected from the basic output blocks according to preset screening criteria; the screening criteria include at least: the absolute value of the basic output components, the compensation priority score of the basic output components, and the saturation risk of the basic output components; wherein, the compensation priority score can be obtained by weighted fusion of "component absolute value" and "corresponding channel error characteristics"; the saturation risk can be characterized by "the ratio of component absolute value to saturation threshold" or "the function value of the distance between the component exponent bit and the FP16 exponent boundary".

[0062] (18) "Top-K Filtering Parameters": refers to the set of parameters used by the Top-K filtering module to determine the number K to be filtered and the basis for filtering.

[0063] (19) "Compensation coefficient parameters": refers to the set of parameters used for residual compensation operations, including but not limited to the linear mapping coefficients of the piecewise function, the bias term, and the weight parameters related to the error characteristics.

[0064] (20) "Residual compensation operation": refers to the compensation calculation performed on the Top-K components to generate compensation components; the residual compensation operation includes at least element-wise multiplication and addition operations, and the Top-K components can be mapped using a preset piecewise function.

[0065] (21) "Compensation component": refers to the output obtained by the residual compensation operation, which is used to merge with the basic output block to obtain the compensation output block.

[0066] (22) "Fusion": refers to the operation of superimposing the compensation component onto the corresponding position of the basic output block to obtain the compensation output block.

[0067] (23) "Compensated output block": refers to the output block obtained after fusing the compensation components, which is used as the output of the current layer and input to the next layer for inference calculation.

[0068] (24) "Parameter storage device": refers to a storage device used to store threshold parameters, Top-K filtering parameters, compensation coefficient parameters and scaling strategy parameters. The device may be on-chip storage or off-chip storage, or a combination of the two.

[0069] (25) "Inference execution device": refers to a computing device used to execute the above method steps, including a matrix calculation module for performing matrix operations and a vector calculation module for performing vector operations, and may include functional modules such as block division, feature calculation, trigger discrimination, compensation and fusion.

[0070] Advantages and positive effects of the present invention:

[0071] This invention addresses the problems of limited dynamic range, accumulated rounding errors, and unstable results in the Ascend NPU edge-side FP16 inference scenario. It proposes a precision compensation method and system based on block dynamic scaling and triggered Top-K residual compensation, which has at least the following advantages and positive effects:

[0072] (1) By calculating statistical features and determining dynamic scaling factors for input blocks and / or their corresponding weight blocks, the range of input values ​​for matrix multiplication or convolution operations can be adaptively made to fall within the preset FP16 safety range, thereby reducing output instability caused by overflow, underflow and rounding errors in end-side FP16 inference.

[0073] (2) By extracting error features on the vector calculation unit and performing trigger discrimination, residual compensation operation is enabled only when the preset compensation conditions are met, thus avoiding the extra overhead caused by full compensation of all components;

[0074] (3) By using Top-K filtering, the compensation calculation is limited to the range of error-sensitive components, and an index set and / or mask set is output to guide the compensation position, thereby achieving selective correction of key components and improving output consistency while controlling the amount of computation and memory access overhead.

[0075] (4) By organizing the main computation on the matrix side and the error analysis / compensation on the vector side in parallel in a collaborative pipeline manner, the compensation computation is minimized in terms of the throughput of the matrix computation unit, thereby balancing numerical stability and execution efficiency in real-time inference scenarios on the edge. Attached Figure Description

[0076] Figure 1This is a flowchart of an FP16 accuracy compensation method for Ascend NPU provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an FP16 precision compensation system for Ascend NPU provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of "blocking strategy and block alignment constraint" in an embodiment of the present invention; Figure 4 This is a schematic diagram of "error feature extraction, trigger discrimination and Top-K screening" in an embodiment of the present invention; Figure 5 This is a schematic diagram of "residual compensation calculation and fusion output" in an embodiment of the present invention;

[0077] Figure 6 This is a timing diagram of "coordinated pipelined execution of matrix calculation and vector compensation" in an embodiment of the present invention. Detailed Implementation

[0078] To enable those skilled in the art to better understand the technical solution of the present invention, the specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments of the present invention are only used to explain the present invention, and not to limit the present invention; in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0079] Example 1: FP16 Precision Compensation System for Ascend NPU

[0080] like Figure 2 As shown in the overall system structure, this embodiment of the invention provides an FP16 accuracy compensation system for Ascend NPU, including an inference execution device 100 and a parameter storage device 300. The inference execution device 100 and the parameter storage device 300 can communicate via wired or wireless connection; wired connection includes, but is not limited to, PCIe, USB, Ethernet, etc., while wireless connection includes, but is not limited to, WiFi, Bluetooth, cellular network, etc. The parameter storage device 300 stores threshold parameters, Top-K filtering parameters, compensation coefficient parameters, and scaling strategy parameters corresponding to each layer, and provides these parameters to the inference execution device 100 as needed. The inference execution device 100 executes the FP16 accuracy compensation method of this invention to complete edge-side inference calculations.

[0081] In one embodiment, the inference execution device 100 is an end-side device or edge-side device that includes an Ascend NPU; the parameter storage device 300 can be on-chip storage, off-chip storage, or external storage device of the end-side device, or it can be a network-side storage device; the inference execution device 100 can load the required parameters into the local cache before inference begins, so as to reduce the parameter access overhead during the inference process.

[0082] like Figure 2As shown in the functional modules of the inference execution device 100, the inference execution device 100 includes the following functional modules:

[0083] (1) Block module 110, used to obtain the input tensor and weight tensor to be performed inference calculation, and divide the input tensor and weight tensor into several input blocks and weight blocks respectively according to a predetermined block strategy;

[0084] (2) Feature calculation module 120, used to calculate statistical features for any input block / weight block of any layer;

[0085] (3) Scaling factor determination module 130, used to determine the dynamic scaling factor based on the statistical characteristics;

[0086] (4) Dynamic scaling module 140, used to perform dynamic scaling processing on the input block and / or its corresponding weight block according to the dynamic scaling factor, so as to obtain the scaled input block and / or the scaled weight block;

[0087] (5) Matrix calculation module 150, used to perform matrix multiplication or convolution operation on the scaled input block and the scaled weight block on the matrix calculation unit of Ascend NPU to obtain the basic output block;

[0088] (6) Error feature extraction module 160, used to extract error features on the vector computing unit of Ascend NPU based on the basic output block and / or the input block;

[0089] (7) Trigger discrimination module 170, used to compare the error feature with a preset threshold to obtain a trigger flag;

[0090] (8) Top-K filtering module 180, used to filter Top-K components from the basic output block when the trigger flag meets the preset compensation conditions;

[0091] (9) Residual compensation module 190, used to perform residual compensation operation on the Top-K components based on the compensation coefficient parameters to obtain the compensated components;

[0092] (10) Fusion module 200, used to fuse the compensation component with the basic output block to obtain the compensation output block;

[0093] (11) Output module 210 is used to output the compensation output block as the current layer output and use the current layer output as the input for the next layer inference calculation until the inference calculation is completed.

[0094] It should be noted that the above module division is only an example of functional division and does not constitute a limitation of the present invention; in actual implementation, some functions in the above modules can be implemented by the same module, or by different modules working together, or by software, hardware, firmware or a combination thereof.

[0095] Contents and organization of parameter storage device 300

[0096] like Figure 2 As shown, the parameter storage device 300 is used to store the parameter set associated with the inference layer and to provide the inference execution device 100 with the threshold, filtering, compensation, and scaling strategy parameters required for runtime. The parameter set includes at least: (1) a threshold parameter 310, used by the error feature extraction module 160 and the trigger discrimination module 170, wherein the threshold parameter includes at least a trigger discrimination threshold. and the saturation threshold used to calculate the saturation ratio. (2) Top-K screening parameter 320, used by Top-K screening module 180 to determine the screening quantity K and screening basis (e.g., absolute value, priority score, weight of saturation risk score, etc.); (3) Compensation coefficient parameter 330, used by residual compensation module 190 to calculate compensation component. The compensation coefficient parameter includes at least the segmented threshold. And the coefficients and bias terms of the piecewise linear mapping In one alternative implementation, the aforementioned compensation coefficient parameters can be determined by offline calibration or training fitting: reference outputs are obtained from the calibration data respectively. With basic output Calculate the residual And obtained through least squares fitting or constrained regression fitting. Then, it is stored by layer or by operator type index; (4) Scaling strategy parameter 340, which is used by scaling factor determination module 130 and dynamic scaling module 140 to determine and execute dynamic scaling processing. The scaling strategy parameter includes at least:

[0097] Target safety amplitude A safe Quantile parameter p, tail clipping parameter ρ (or used to calculate ρ) eff ρ0, ρ1, ρ2 and their upper and lower bounds ρ min ρ max The following parameters are also considered: fusion weight β (or its calculation parameters β0, β1, etc.), exponential safety margin μ, scaling sensitivity coefficient κ, basic scaling exponent n0, allowable scaling exponent range [n_min, n_max], and weight coefficient w for selecting the objective function. risk w mantissa w trigger And its index relationships by level / by operator type.

[0098] To avoid ambiguity, the meanings of the above symbols are explained as follows (not restrictive):

[0099] This represents the target safety amplitude / target scale corresponding to the preset FP16 safety range.

[0100] Quantile parameter (used for calculation) of quantiles );

[0101] This represents the tail clipping factor, used to construct the effective amplitude. To reduce overscaling caused by outliers;

[0102] This represents the index safety margin, used to constrain the index range and avoid approaching the FP16 index boundary.

[0103] This represents the scaling sensitivity coefficient, which is used to feed back the comparison result of error characteristics and threshold to adjust the scaling intensity.

[0104] Indicates the base scaling index; Indicates the allowable range of the scaling exponent, used to constrain the scaling factor. The value of ; This represents the weighting coefficient in the objective function selection rule, used to make trade-offs between out-of-bounds risk, effective tail loss, and trigger compensation overhead.

[0105] Represents the dynamic tail clipping factor ρ eff The calculation parameters (ρ0, ρ1, ρ2) and their upper and lower bounds ρ min ρ max ; indicates the fusion weight β, used in A tail Weighted fusion is performed between r to construct the effective fusion amplitude A. fuse E prev I(·) represents the error characteristics or trigger result feedback of the previous block; I(·) represents the indicator function; A fuse Indicates the effective amplitude of fusion; A tail This indicates the effective amplitude after tail cropping.

[0106] In one implementation, the parameter set is stored by layer index, i.e., each layer... For at least one set of threshold parameters, Top-K filtering parameters, compensation coefficient parameters, and scaling strategy parameters, it can be expressed as:

[0107]

[0108]

[0109] During inference execution, read according to the current layer number. This completes the trigger detection and compensation calculation.

[0110] In another implementation, the parameter set is stored indexed by operator type, meaning that operators of the same type share at least some parameters to reduce the parameter size, for example, for... shared Alternatively, a set of compensation coefficient parameters can be shared; when more precise control is required, a hybrid indexing method using a combination of "operator type + layer number" can be used.

[0111] Hardware Motivation Explanation

[0112] In this embodiment, the Ascend NPU can be a Da Vinci architecture processor with the ability to collaboratively execute matrix-type and vector-type computation units; for example, any model in the Ascend series can be selected (such as Ascend 310, Ascend 910, or their subsequent models, which are all non-limiting examples). The Da Vinci architecture typically adopts a heterogeneous computing structure with collaborative matrix-type and vector-type computation units: the matrix-type computation units are suitable for performing high-throughput matrix multiplication and convolution intensive multiplication-addition operations, while the vector-type computation units are suitable for performing element-wise multiplication-addition, reduction statistics, and threshold comparison operations. If the compensation operation is directly incorporated into the main matrix computation path, it is easy to occupy matrix-type computation resources and introduce additional synchronization and memory access overhead, thereby reducing edge throughput and amplifying latency jitter. To this end, the present invention places dynamic scaling before the main matrix computation to reduce the risk of overflow / underflow, and places error feature extraction, trigger discrimination, and Top-K residual compensation on the vector-type computation unit side, thereby achieving selective correction of low-precision errors without significantly affecting the throughput of the main matrix computation.

[0113] In this embodiment, when performing inference, the inference execution device 100 performs the following steps for any layer in the network that requires precision compensation:

[0114] Step 101 (corresponding to) Figure 1 Blocking process 110): Obtain the input tensor and weight tensor and divide them into blocks. In this embodiment, the input tensor and weight tensor are divided into a set of input blocks and a set of weight blocks, respectively, and a correspondence between the input blocks and weight blocks is established to form a set of block pairs. This step corresponds to... Figure 1 The block-based process 110 in the middle can be... Figure 2The block segmentation module 110 in the system is executed. It acquires the input tensor and weight tensor to be used for inference computation, and divides the input tensor and weight tensor into a set of input blocks and a set of weight blocks respectively according to a predetermined block segmentation strategy. In this embodiment, the block segmentation module 110 performs block processing on the current layer input tensor to obtain an input block set; simultaneously, it performs block processing on the current layer weight tensor to obtain a weight block set. The block partitioning strategy includes at least block dimensions, block sizes, and block alignment constraints. The block alignment constraints are used to ensure that: firstly, when the system matrix calculation module 150 performs matrix multiplication or convolution operations on the matrix calculation unit, the dimensions of each block meet the hardware execution requirements; secondly, the data layout of the basic output blocks matches the data layout of subsequent vector-side statistical feature calculation, error feature extraction, and element-by-element compensation operations, thereby providing a consistent data organization form for subsequent trigger discrimination and Top-K selective compensation; furthermore, the block alignment constraints are also used to match the storage layout of the basic output blocks in the cache with the continuous memory access method of the vector calculation unit, so as to reduce the overhead of cross-row skip memory access and data rearrangement in the process of vector-side reading, statistics, and element-by-element compensation, thereby improving the throughput stability when executed on the Ascend NPU edge.

[0115] Step 102 (corresponding to) Figure 1 The statistical feature calculation process 120 and scaling factor determination process 130): Calculate the block statistical features and determine the dynamic scaling factor. Corresponding to... Figure 1 The statistical feature calculation process 120 and the scaling factor determination process 130 can be executed by the system's statistical feature calculation module 120 and scaling factor determination module 130, respectively. For the input block and its corresponding weight block corresponding to the block index in any network layer, the statistical features of the input block and / or its corresponding weight block are calculated, and a dynamic scaling factor is determined based on these statistical features. In this embodiment, the statistical feature calculation module 120 calculates statistical features characterizing the numerical distribution of the input block and / or its corresponding weight block. These statistical features include, but are not limited to, maximum absolute value, mean, variance, root mean square value, absolute value quantiles, and exponential range representations. The scaling factor determination module 130 determines the dynamic scaling factor based on the statistical features and scaling strategy parameters, ensuring that the numerical range of the input block and / or its corresponding weight block after dynamic scaling falls within a preset FP16 safety range, thereby reducing the risk of underflow, overflow, and rounding error amplification in subsequent matrix multiplication or convolution operations.

[0116] In one optional implementation, to facilitate efficient implementation on the Ascend NPU side, the dynamic scaling factor can be expressed as an integer power of two, i.e. It is an integer and limited to a preset power range. Within this framework, scaling operations can be approximated using shifting or fast multiplication. Let the data block be B (the input block or weighted block), and its absolute value statistics include the maximum absolute value. Root mean square and absolute value quantiles Let the target safety amplitude corresponding to the preset FP16 safety range be denoted as . (Can be obtained from offline calibration), then the candidate scaling factor value can be generated in any of the following ways: (1) Maximum absolute value scaling:

[0117]

[0118] (2) Root mean square scaling:

[0119]

[0120] (3) Quantile scaling:

[0121]

[0122] (4) Tail clipping and scaling: To avoid overscaling due to a very small number of outliers and loss of the effective tails of most components, the effective amplitude can be constructed first.

[0123]

[0124] in This is the tail trimming factor; then take...

[0125]

[0126] (5) Compensation-based perceptual multi-statistical fusion scaling (preferred method of the present invention): Based on tail pruning, root mean square statistics are introduced to characterize the main distribution energy, and the Top-K budget and the error characteristics of the previous block are fed back into the adaptive adjustment of pruning intensity and fusion weight. Let the block data be x (input block or weight block).

[0127] , , ,in Constructing the dynamic tail clipping factor.

[0128] and obtained Further, the main energy and tail amplitude are fused using fusion weights β∈[0,1]. Based on this, candidate scaling indices can be selected.

[0129] scaling factor .in The interval boundaries are configurable parameters and can be stored by layer or shared by operator type; E prev The error characteristics or triggering results obtained from the previous block can be provided by the vector side, thus forming a closed-loop collaboration of scaling-triggering-compensation.

[0130] By using tail-pruning scaling, rare extreme outliers can be left for subsequent trigger-based Top-K residual compensation mechanisms to handle, thereby reducing the loss of effective accuracy caused by over-scaling while ensuring overall numerical stability. Among the candidate scaling indices n (or candidate scaling factors s) generated in the above manner, those that ensure the scaled block numerical range falls within a preset FP16 safety range and meets the requirement of retaining effective tails can be selected as the dynamic scaling factors. When candidates are generated by multiple methods, the candidate scaling indices obtained from each method can be uniformly projected onto... After deduplication, a candidate set is formed, and the final dynamic scaling factor is selected according to the above criteria.

[0131] In a further optional implementation, in addition to generating candidates based on amplitude statistics, FP16 exponent range constraints, compensation trigger thresholds, and Top-K compensation budget parameters can be introduced to generate candidate scaling indices or select from the candidate set, giving the scaling strategy constraint-driven and compensation-aware parameterized characteristics. For example, at least one of the following construction methods can be used: (A) Construction of exponent range constraints: Determine the scaling index based on the exponent range of the block data in FP16 representation and the preset safety index interval, and trim it to... (B) Construction of compensation trigger threshold awareness: Adjust the scaling exponent based on the relative relationship between error characteristics and trigger threshold to reduce trigger frequency or compensation magnitude; (C) Construction of Top-K budget awareness tail retention: Set quantiles and construct effective magnitudes based on the tail proportion that Top-K budgets can cover to avoid overscaling for a few outliers; (D) Objective function selection: Construct a cost function within the candidate set based on exponent out-of-bounds risk, effective tail loss, and end-side triggering cost estimation, and select the optimal scaling exponent. Through these methods, a configurable trade-off can be achieved between numerical stability, accuracy, and end-side cost.

[0132] Step 103 (corresponding to) Figure 1 The dynamic scaling process 140): Perform dynamic scaling on the input block and / or its corresponding weight block. Figure 1The dynamic scaling process 140 can be executed by the system dynamic scaling module 140. Dynamic scaling is performed on the input block and / or its corresponding weight block according to the dynamic scaling factor to obtain the dynamically scaled input block and weight block. In this embodiment, the dynamic scaling module 140 performs scaling on the input block and / or its corresponding weight block according to the dynamic scaling factor, while ensuring that the desired output scale of matrix multiplication or convolution operations remains unchanged. Specifically, dynamic scaling can be performed in one of the following ways: (1) scaling the input block by multiplying it by a dynamic scaling factor, and simultaneously scaling the corresponding weight block by multiplying it by the reciprocal of the dynamic scaling factor; (2) scaling the weight block by multiplying it by a dynamic scaling factor, and simultaneously scaling the corresponding input block by multiplying it by the reciprocal of the dynamic scaling factor; (3) scaling the input block by multiplying it by a dynamic scaling factor only, keeping the weight block unchanged, and performing reverse scaling or equivalent scale correction on the output in the subsequent fusion stage; (4) scaling the weight block by multiplying it by a dynamic scaling factor only, keeping the input block unchanged, and performing reverse scaling or equivalent scale correction on the output in the subsequent fusion stage. Through the above processing, the adaptive adjustment of the numerical dynamic range can be achieved without changing the target scale of the operation, thereby providing more stable input numerical conditions for the main computation of the matrix computation unit.

[0133] Step 104 (corresponding to) Figure 1 Matrix multiplication / convolution main computation flow 150): Matrix multiplication / convolution operations are performed in the matrix computation unit to obtain the basic output blocks. .correspond Figure 1 The matrix calculation process 150 in the system can be executed by the system matrix calculation module 150. On the matrix calculation unit of the Ascend NPU, matrix multiplication or convolution operations are performed on the dynamically scaled input block and its corresponding weight block to obtain the basic output block. In this embodiment, the matrix calculation module 150 performs matrix multiplication or convolution operations on the dynamically scaled input block and its corresponding weight block on the matrix calculation unit to obtain the basic output block; the basic output block is the output result before residual compensation is performed. The basic output block is written to a predetermined buffer area for subsequent error feature extraction, trigger discrimination, and residual compensation operations performed by the vector calculation unit, thereby forming a collaborative processing link between the matrix calculation unit and the vector calculation unit.

[0134] Step 105 (corresponding to) Figure 1 Error feature extraction process 160 and threshold comparison process 170): Extract error features and compare them with the threshold to obtain the trigger flag. Figure 1The error feature extraction process 160 and the threshold comparison process 170 can be executed by the system error feature extraction module 160 and the trigger discrimination module 170, respectively. On the Ascend NPU's vector computation unit, error features characterizing the risk of low-precision errors are extracted based on the basic output block, and optionally combined with auxiliary features extracted from the input block corresponding to the basic output block to determine the error features; the error features are compared with the threshold parameters provided by the parameter storage device 300 to obtain a trigger flag.

[0135] Specifically, the error feature extraction module 160 divides the basic output into blocks on the vector calculation unit. Calculate at least one of the following characterization quantities, and generate an error eigenvalue E according to a preset combination rule:

[0136] (1) Energy characterization quantity Used to characterize the amplitude / energy level of the basic output blocks, for example, it can be taken as... , Or its normalized form;

[0137] (2) Dispersion characterization Used to characterize the degree of dispersion of the basic output block distribution, such as available variance, coefficient of variation, or quantile interval;

[0138] (3) Saturation ratio characterization quantity : Used to characterize the proportion of the basic output block that saturates, for example, it can be defined as

[0139]

[0140] Where N is the number of basic output blocks of elements. Based on the output components, This is the saturation threshold;

[0141] (4) (Optional) Auxiliary characterization : Obtained from the statistical characteristics of the input blocks (and / or their corresponding weight blocks) corresponding to the basic output blocks, used to enhance the perception of abnormal input distributions.

[0142] The preset combination rules include at least one of the following:

[0143] (a) Normalize the above representations and then sum them by weight, for example...

[0144] in Preset weight parameters;

[0145] (b) After concatenating the above-mentioned representations into vectors, a linear or nonlinear mapping is performed to obtain the error feature value E. The discrimination module 170 is then triggered to compare the error feature value E with the threshold parameter. (Can be configured and stored in parameter storage device 300 by layer / type) Compare and generate trigger flag; the trigger flag is used to indicate whether the current basic output block meets the preset compensation conditions.

[0146] Parameter source explanation: The threshold parameter saturation threshold and combined weight parameters It can be configured by layer or by operator type, and can be obtained through offline calibration, training optimization, or empirical settings, and stored in parameter storage device 300 for inference execution device to access on the edge; wherein, saturation threshold It can be determined by the amplitude / exponential range mapping corresponding to the preset FP16 safety range or obtained by offline calibration.

[0147] Step 106 (corresponding to) Figure 1 (Compensation condition determination process 175, Top-K screening process 180, and residual compensation calculation process 190): Step 106: Determine whether the compensation condition is met based on the trigger flag; if met, screen the Top-K components and calculate the compensation component ∆. Corresponding to... Figure 1 The compensation condition determination 175, Top-K screening process 180, and residual compensation calculation process 190 can be executed by the system determination unit 175, the Top-K screening module 180, and the residual compensation module 190, respectively; if the conditions are not met, compensation is skipped. When the trigger flag meets the preset compensation conditions, the Top-K screening module 180 selects K error-sensitive components from the basic output block according to the Top-K screening parameters to form a Top-K candidate set, and outputs the index set corresponding to the Top-K components. AND / or mask set , used to indicate the calculation location of residual compensation operations.

[0148] In this embodiment, the selection criteria for Top-K components include at least one of the following or a combination thereof: (1) the absolute value of the basic output component; (2) the compensation priority score of the basic output component; and (3) the saturation risk of the basic output component. The compensation priority score can be obtained by weighted fusion of the "component absolute value" and the "channel-level error risk characteristic" (e.g., the error characteristic value obtained in step 105 or its channel aggregation value); the saturation risk can be characterized by the "ratio of the component absolute value to the saturation threshold" or the "function value of the distance between the component exponent bit and the FP16-represented exponent boundary".

[0149] The system residual compensation module 190 adjusts the compensation coefficient parameters according to the... The indicated locations are used to perform residual compensation calculations to obtain the compensated components. To calculate the residuals, reference / estimated values ​​are obtained for the selected Top-K locations. And calculate the residual components.

[0150]

[0151] in The basic output component; It can be obtained by any of the following methods (not limited): a reference output value obtained by recalculating the local calculation corresponding to the Top-K position with a higher precision path (e.g., FP32 / BF16 or high precision accumulation), or an estimated value obtained by error model / lookup table / approximate estimation.

[0152] In one alternative implementation, the residual compensation operation uses a piecewise linear mapping method to calculate the compensation component. Taking any selected residual component d as the object, its compensation amount is calculated as shown in formula (1): Formula (1):

[0153]

[0154] in, This represents the corresponding compensation amount; τ is the segmented threshold. The coefficients and bias terms of the piecewise linear mapping are determined by the compensation coefficient parameters obtained from the offline calibration fitting and can be stored layer by layer in the parameter storage device 300. The calculation is performed for any Top-K position. Then, it is merged into the corresponding position of the basic output block to obtain the compensated output block. By using piecewise linear mapping, the error in different numerical ranges can be differentially corrected while keeping the compensation calculation cost low.

[0155] Reference / estimated value acquisition method: In one optional implementation, the reference value is used as control-side overhead. Only for the aforementioned set of indexes The reference output value is obtained by specifying the Top-K positions, rather than by recalculating the entire basic output block with high precision. Specifically, the local computation corresponding to the Top-K positions can be recalculated using a higher-precision path to obtain the reference output value (e.g., by using a higher-precision data type or a higher-precision accumulation method to perform local matrix multiplication / convolution operations), and the recalculation result is used as the reference value. Alternatively, the reference values ​​for the Top-K positions can be approximated based on the error estimation model or lookup table obtained from offline calibration / training. This allows the residual components to be obtained with only a small increase in computational and storage overhead. This is used for subsequent residual compensation calculations.

[0156] Step 107 (corresponding to) Figure 1 Write-back fusion process 200): Write back the compensation component Δ and combine it with the base output in blocks. The fusion yields a compensated output block Y, corresponding to Figure 1 The fusion process 200 can be executed by the system fusion module 200. The compensated component is fused with the basic output block to obtain a compensated output block; otherwise, the basic output block is directly used as the compensated output block. In this embodiment, the fusion module 200 writes the compensated component back to the corresponding position of the basic output block and outputs the compensated result as the compensated output block. For components not selected as Top-K, their output remains unchanged from the original value of the basic output block, thereby achieving selective compensation for error-sensitive components. When the trigger flag does not meet the preset compensation conditions, the Top-K filtering and residual compensation calculation are skipped, and the basic output block is directly output as the compensated output block. .

[0157] Step 108 (corresponding to) Figure 1 The process includes: outputting the current block result (flow 210), determining whether all block divisions of the current layer are complete (flow 214), and determining whether all layer inferences are complete (flow 215). The output module 210 writes the compensated output block to the corresponding position in the current layer's output tensor; subsequently, the decision logic executes... Figure 1 The judgment in process 214 is as follows: If the result of "Has the reasoning of all blocks in the current layer been completed?" is "No", then update the block index b and return to step 102 to continue processing the next block pair; if the result is "Yes", then continue to be executed by the judgment logic. Figure 1 The judgment in process 215 is as follows: It determines whether "all layers of inference have been completed?" (i.e., whether there is a next layer that needs to be inferred). If the result is "no", the current layer's output tensor is used as the input for the next layer, and the process returns to step 101. If the result is "yes", the inference calculation ends. The above write-back and judgment process can be implemented by the inference execution device 100 under the control and scheduling of the processor / Ascend NPU by executing the program instructions of the output module 210 and the end judgment unit (including the judgment instructions corresponding to processes 214 / 215).

[0158] Optional implementation methods for collaborative pipelined execution

[0159] like Figure 6As shown, in one optional embodiment, to reduce the additional latency introduced by accuracy compensation and improve edge inference throughput, the inference execution device 100 coordinates the processing of the matrix calculation unit and the vector calculation unit to achieve pipelined execution. Specifically, the matrix calculation module 150 performs matrix multiplication or convolution operations on the current input block and the current weight block in the matrix calculation unit to generate the current basic output block; at the same time, the error feature extraction module 160, the trigger discrimination module 170, and the residual compensation module 190 perform error feature extraction, trigger discrimination, and selective residual compensation operations on the previous basic output block in the vector calculation unit, thereby enabling the matrix-side main calculation and the vector-side compensation calculation to be performed in parallel.

[0160] To ensure the correct completion of the aforementioned pipelined execution and reduce data transfer overhead, this implementation further defines block alignment constraints and data layout matching requirements: the input block, weight block, and basic output block must satisfy a predetermined alignment relationship, ensuring that the block output layout of the matrix calculation unit is consistent with the element-wise operation layout of the vector calculation unit. Furthermore, the reading of the basic output block and the writing back of the compensation component on the vector side can be completed within a predetermined buffer area, avoiding additional latency caused by frequent cross-level data transfer. Through this collaborative pipelined execution method, the operational stability and execution efficiency of edge-side inference can be further improved while ensuring the compensation effect.

[0161] Application Examples

[0162] The following is a specific application example illustrating the implementation process of the method of the present invention in performing FP16 precision compensation on a typical matrix multiplication layer on the Ascend NPU. It should be understood that this application example is only for explaining the present invention and does not constitute a limitation thereof; without departing from the spirit of the present invention, those skilled in the art can apply the present invention to other matrix operators such as convolutional layers and attention projection layers.

[0163] Application scenarios and objects

[0164] This example uses a matrix multiplication layer in a neural network as an example. This layer multiplies and adds input features to weight parameters to generate output features. During FP16 inference on the edge, when the numerical distribution of the input features fluctuates with changes in the scene, the basic output of the matrix multiplication may have insufficient dynamic range or amplified rounding errors, leading to accumulated output bias in this layer and adversely affecting subsequent layers. To address this issue, this example introduces a processing chain of "block dynamic scaling + error feature-triggered discrimination + Top-K selective residual compensation" during the inference execution of this matrix multiplication layer. This corrects error-sensitive components while controlling additional computational overhead, improving output stability.

[0165] Parameter setting method (exemplary explanation)

[0166] Parameter Source and Setting Principles: In this example, when the inference execution device 100 performs the matrix multiplication layer inference, it reads and uses the following parameters from the parameter storage device 300: threshold parameters, Top-K filtering parameters, compensation coefficient parameters, and scaling strategy parameters. These parameters can be set in the following manner (the following is an illustrative example; the specific setting method and values ​​can be adjusted according to the model structure and deployment constraints):

[0167] 1) Threshold parameter: Used to trigger the discrimination module to compare error features and generate a trigger flag. The threshold parameter can be determined through offline calibration, for example, by statistically analyzing the distribution of error features on a small number of calibration samples and selecting a threshold that keeps the trigger rate within a preset range as the threshold parameter.

[0168] 2) Top-K Filtering Parameters: These parameters are used to limit the number of components to be selectively compensated and to specify the criteria for Top-K filtering. The Top-K filtering parameters can be set based on the edge computing power budget and the expected latency limit. When it is necessary to prioritize meeting real-time constraints, a smaller K value can be set to reduce the compensation calculation overhead.

[0169] 3) Scaling strategy parameters: These determine the calculation rules for the dynamic scaling factor. The scaling strategy parameters may include a preset FP16 safety range, an exponential range mapping rule, and a candidate set of scaling factors to ensure that the numerical range after block scaling meets the low-precision stability requirements. In one exemplary implementation, the preset FP16 safety range can be determined by offline calibration: selecting calibration samples covering the target scene, statistically analyzing the exponential distribution range for the input block and the weight block respectively, and reserving a safety margin at the upper and lower boundaries of the exponential distribution; simultaneously, setting an effective mantissa retention threshold based on the target task's numerical accuracy requirements to avoid loss of effective accuracy caused by over-scaling. The resulting exponential range and safety margin together constitute the preset FP16 safety range.

[0170] 4) Compensation coefficient parameters: These are parameters used for residual compensation calculations. They can be used to determine the segment threshold, linear mapping coefficients, and bias terms of piecewise linear mapping. The compensation coefficient parameters can be obtained through offline fitting or rule setting to reduce the output deviation after compensation.

[0171] Example parameter table: Table 1 Example parameter description

[0172] (1) Block size parameter: Number of block elements N = 4096 (e.g., based on the number of elements after the output block is expanded). (2) Top-K parameter: K = 32 (e.g., residual compensation is performed on only 32 components per block). (3) Quantile corresponding to compensation budget: p K= 1 − K / N = 1 − 32 / 4096 ≈ 0.9922 (i.e., 99.22% quantile). (4) Scaling factor form: s = 2^n, where n∈[n min , n max Example: take n min = −8,n max = 8. (5) Target safety amplitude (FP16 safety range corresponding): Example: A safe = 8192 (used to allow for margin in the exponent and effective tail). (6) Tail clipping factor: Example: take R = 1.30 (used to construct the effective amplitude m) eff (7) Zero division protection constant: Example: ε = 1e−6. (8) Trigger threshold: Example: θ = 0.10 (compensation is triggered when the error characteristic E exceeds this threshold). (9) Example definition of error characteristic E: Example: E = d max / (max(|Y ref |)+ε), where d max Y represents the maximum residual amplitude within the block. ref The reference value or high-precision estimate obtained by the vector computation unit (one of the example definitions, not limited). (10) Piecewise linear compensation parameters: In the example, τ = 64; when |d| ≤ τ, g(|d|) = a1·|d| + b1 is used, in the example a1 = 0.70, b1 = 0; when |d| > τ, g(|d|) = a2·|d| + b2 is used, in the example a2 = 1.00, b2 = 0. (11) Compensation sign: In the example, Δ = sign(d)·g(|d|), where d = Y ref - Y0, where Y0 is the basic output block of the matrix calculation unit.

[0173] (12) Example of scaling application method: In order to keep the overall scale unchanged, the example uses a combination of input block scaling and weight block inverse scaling: X′ = s·X, W′ = (1 / s)·W, so that X′·W′ and X·W are equivalent in the ideal real number domain (one of the example methods, not limited).

[0174] Execution process

[0175] In this example, such as Figure 3 As shown, the input tensor and weight tensor are first divided into input blocks X according to a predetermined block partitioning strategy. (i) With weighted block W (j) And satisfy alignment constraints such as dimension, storage layout, stride, and continuity; subsequently, as Figure 1 , Figure 4 , Figure 5As shown, for any input block and its corresponding weight block of the matrix multiplication layer, the inference execution device 100 performs accuracy compensation according to the following process:

[0176] (1) Statistical feature calculation and dynamic scaling factor determination: The feature calculation module calculates statistical features for the input block and / or its corresponding weight block; the scaling factor determination module determines the dynamic scaling factor based on the statistical features and scaling strategy parameters.

[0177] (2) Dynamic scaling processing: The dynamic scaling module performs dynamic scaling processing on the input block and / or its corresponding weight block according to the dynamic scaling factor to obtain the scaled input block and / or scaled weight block, and ensures that the expected output scale of the matrix multiplication operation remains unchanged.

[0178] (3) Matrix main calculation: The matrix calculation module performs matrix multiplication on the scaled input block and the scaled weight block in the matrix calculation unit to obtain the basic output block; the basic output block is the output result before compensation.

[0179] (4) Error feature extraction and trigger discrimination: The error feature extraction module extracts error features based on the basic output block and / or input block on the vector calculation unit; the trigger discrimination module compares the error features with the threshold parameter to obtain the trigger flag, so as to determine whether the preset compensation condition is met.

[0180] (5) Top-K filtering and residual compensation: When the trigger flag meets the preset compensation conditions, the Top-K filtering module filters the Top-K components from the basic output blocks according to the Top-K filtering parameters; the residual compensation module performs residual compensation operation on the Top-K components according to the compensation coefficient parameters to obtain the compensation components. The residual compensation operation can use the aforementioned piecewise linear mapping method to calculate the compensation amount. The calculation method is shown in formula (1).

[0181] (6) Fusion Output: The fusion module writes the compensation component back to the corresponding position of the basic output block to obtain the compensation output block; for the unfiltered component, its output remains unchanged from the original value of the basic output block. If the trigger flag does not meet the preset compensation conditions, the Top-K filtering and residual compensation operation are skipped, and the basic output block is directly output as the compensation output block.

[0182] (7) Output: The output module outputs the compensation output block as the output of the matrix multiplication layer and as the input for the inference calculation of the subsequent layer (if there is a subsequent layer).

[0183] Numerical demonstration of a single block partitioning

[0184] The following is a complete demonstration of a "single block pair" (i.e., a set of input blocks and their corresponding weight blocks) to illustrate the computable process of dynamic scaling, trigger discrimination, Top-K filtering, and residual compensation on a block pair according to the present invention. In actual inference, the input tensor and weight tensor are first divided into several input blocks and several weight blocks according to a predetermined block strategy, and a set of block pairs is established. Then, the same processing flow as in this demonstration is executed sequentially on the block pair set according to the index. The differences between different blocks are mainly reflected in the block data itself and its corresponding statistical characteristics, scaling factors, and trigger results, while the processing steps and computation chain remain consistent.

[0185] (1) Given the block and Top-K budget

[0186] Suppose the number of elements after the current output block is expanded is N = 4096, and the Top-K parameter is K = 32, then

[0187] p K = 1 − K / N = 1 − 32 / 4096 ≈ 0.9922.

[0188] (2) Statistical characteristics and effective amplitude construction, using compensated sensing tail clipping

[0189] Suppose that the calculation for the input block (or weight block) yields:

[0190] The maximum absolute value m = 18000;

[0191] 99.22 percentile q = 9000.

[0192] Root mean square r = 11200.

[0193] For example, take ρ0=1.25, ρ1=6, ρ2=0, and E prev If ≤θ is false (I=0), then

[0194] In one example, candidate scaling factors can be constructed based on the maximum absolute value, root mean square, and quantile, respectively, and combined with ρ. eff Tail clipping is performed; further, the main energy and tail amplitude are fused using β to obtain the fused effective amplitude A. fuse In this demonstration example, β = 0.15, then A fuse =(1−β)·A tail +β·r=0.85×11700+0.15×11200=11625.

[0195] (3) Calculate the scaling exponent n and the scaling factor s

[0196] Example: Take Asafe = 8192, ε = 1e−6, then

[0197] n = clip( round( log2( A safe / (m_eff+ε) ) ), n_min, n_max )

[0198] = clip( round( log2(8192 / 11700) ), −8, 8 )

[0199] = clip( round( log2(0.700) ), −8, 8 )

[0200] ≈ clip( round(−0.514), −8, 8 ) = −1.

[0201] Therefore, the scaling factor s = 2^n = 2^{−1} = 0.5.

[0202] (4) Apply scaling (using "input scaling, weighted inverse scaling" to maintain the overall scale)

[0203] Input block X′ = s·X = 0.5·X; Weight block W′ = (1 / s)·W = 2·W.

[0204] If the maximum value of the original input block is about 18000, then it will be about 9000 after scaling; if the maximum value of the original weight block is about 0.9, then it will be about 1.8 after scaling (both remain within the safe range of the example FP16).

[0205] (5) Matrix calculation yields the basic output block Y0

[0206] The matrix computation unit performs matrix multiplication / convolution in FP16 to obtain the basic output block Y0.

[0207] Example of taking some output components (example only):

[0208] Y0[i1] = 12500, Y0[i2] = −15000, and max(|Y0|)≈16000 within the block.

[0209] (6) The vector calculation unit provides reference values ​​and calculates residuals.

[0210] The vector computation unit obtains a reference value / high-precision estimate Y for the candidate components (or performs lightweight estimation on the blocks). ref Example:

[0211] Y refIf [i1] = 12540, then d[i1] = Y ref [i1] ​​− Y0[i1] = 40;

[0212] Y ref If [i2] = −13400, then d[i2] = 1600.

[0213] And set the maximum amplitude d of the residual within the block. max ≈ 2000, max(|Y ref If |)≈16000, then the error characteristic example is:

[0214] E = d max / (max(|Y ref |)+ε) ≈ 2000 / 16000 = 0.125.

[0215] In a simplified implementation, the error characteristic can be taken as the normalized residual magnitude.

[0216] ;

[0217] In other implementations, the error characteristics can also be obtained by combining characteristic quantities such as energy, dispersion and saturation ratio as described in step 105 (not limited).

[0218] (7) Triggering judgment

[0219] In this example, the trigger threshold is set to θ = 0.10. Since E = 0.125 > 0.10, the trigger flag is "Yes", and the Top-K compensation process begins.

[0220] (8) Top-K screening

[0221] Sort the components in the block according to |d| (or equivalent error index), and select the Top-K set K. set (K=32).

[0222] In the example, both i2 (|d|=1600) and i1 (|d|=40) may enter the Top-K (this is just an example; the actual ranking depends on the sorting results).

[0223] (9) Calculation of Δ by piecewise linear compensation

[0224] For example, take τ = 64, a1 = 0.70, b1 = 0, a2 = 1.00, b2 = 0, and define Δ = sign(d)·g(|d|).

[0225] For i1: |d|=40 ≤ τ, therefore g(|d|)=0.70×40=28, Δ[i1]=+28, after compensation

[0226] Y[i1] = Y0[i1] + Δ[i1] = 12500 + 28 = 12528 (the residual decreases from 40 to 12).

[0227] For i2: |d|=1600 > τ, therefore g(|d|)=1.00×1600=1600, Δ[i2]=+1600, after compensation

[0228] Y[i2] = Y0[i2] + Δ[i2] = −15000 + 1600 = −13400 (consistent with the reference value in the example).

[0229] (10) Fusion output and entry into the next layer

[0230] Write back and merge the Top-K set to obtain the compensated output block Y. The remaining components that did not enter the Top-K remain unchanged at Y0, thus obtaining the current layer output block as the input of the subsequent layer or the final output.

[0231] Effect description

[0232] Through the above execution process, this example selectively corrects error-sensitive components only in blocks where the trigger judgment is true, without performing full compensation on the basic output blocks. This improves the numerical stability of FP16 inference output and reduces output fluctuations caused by the accumulation of low-precision rounding errors, while controlling additional computation and memory access overhead. Specifically, this invention reduces the risk of overflow / underflow and loss of effective mantissa through "block dynamic scaling," limits compensation overhead to a small number of high-risk blocks through "error feature trigger judgment," and restricts the compensation operation to element-wise multiplication and addition / piecewise linear mapping calculation of K components through "Top-K residual compensation," making the additional overhead approximately linearly related to K, suitable for real-time inference scenarios on the edge. To facilitate understanding and verification by peers, this embodiment further provides a set of simulation comparison results to illustrate the feasibility and effects of the invention (not limiting).

[0233] Simulation Verification and Effect Comparison (Non-Limited): To verify the feasibility and effectiveness of this invention, the following comparison settings were used for simulation: A single output block of a matrix multiplication layer was taken as the object, the number of elements after the output block was expanded was N=4096, the Top-K parameter was K=32, and the FP32 calculation result was used as the reference value. The comparison methods include: (a) the baseline scheme: directly using FP16 to perform matrix multiplication / convolution to obtain the output (no scaling, no compensation); (b) the comparison scheme: using static scaling (only based on a single scaling factor based on the maximum absolute value) and then performing FP16 matrix calculation (no trigger compensation); (c) the scheme of this invention: using dynamic scaling + error feature trigger discrimination + Top-K selective residual compensation.

[0234] The evaluation metrics should include at least: normalized residual magnitude. The parameters include: overflow / saturation ratio within a block (the percentage of elements whose absolute value exceeds the saturation threshold), trigger rate (the percentage of blocks that are correctly triggered), and compensation ratio (the percentage of components actually compensated within the triggered block, theoretically approximately K / N). Under a set of theoretical example data (consistent with the "Single Block Numerical Demonstration," but not limited), the following comparative results can be obtained:

[0235] (1) Comparison of error characteristics E (block level) Under the baseline scheme, the example takes the maximum magnitude of the residual within the block. ,and

[0236] but ,

[0237] Higher than the trigger threshold

[0238] The conditions for triggering compensation are easier to meet. Under a static scaling scheme, because scaling reduces the risk of dynamic range errors, examples can be made... If it drops to approximately 1400 (unrestricted), then While the error characteristics are reduced, they may still exceed the threshold in some blocks. In this invention, the triggered block only performs selective residual compensation on the Top-K components. In the example, for the critical component, the residual can be compensated from 1600 to 0, and from 40 to 12. Meanwhile, for the remaining Top-K components not shown, the residual magnitude is typically compressed by piecewise mapping (not limited). In this example block, after compensation, the maximum residual magnitude within the block can be reduced to... (Not limited) then

[0239] The output is significantly lower than the threshold, thus improving output stability.

[0240] (2) Comparison of saturation / overflow risks (block-level example) Let the saturation threshold be... (Can be determined by the FP16 exponential boundary and safety margin), statistics The saturation ratio is defined as the percentage of elements. In the example, the saturation ratio of the baseline scheme can be approximately 2.0% (not limited); after static scaling, it can be reduced to approximately 1.0% (not limited); in the scheme of this invention, due to the reduction of the risk of exceeding the limit by dynamic scaling and the triggering of compensation for high-risk components, the saturation ratio can be further reduced to approximately 0.4% (not limited).

[0241] (3) Comparison of Trigger Rate and Compensation Overhead (Intra-Layer Statistical Example) The trigger rate (trigger determination is based on the true percentage) is statistically analyzed for multiple blocks in the same layer. In the example, the trigger rate of the baseline scheme is approximately 35% (not limited), the static scaling scheme is approximately 18% (not limited), and the scheme of this invention is approximately 6% (not limited). When a trigger occurs, the number of compensation components is constrained by Top-K, and the compensation ratio of a single block is approximately... Therefore, the total additional element-wise multiply-accumulate overhead can be estimated at the level of “trigger rate × K” and kept under control.

[0242] Therefore, the present invention can simultaneously achieve the following in theoretical examples: reducing error characteristic E, reducing saturation risk, and limiting compensation overhead to a small proportion, thereby obtaining more robust FP16 inference output on the edge NPU (non-limiting).

[0243] In view of the description and exemplary embodiments of the invention disclosed herein, other embodiments of the invention will be apparent to those skilled in the art. These descriptions and embodiments are considered as examples only, and any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for FP16 accuracy compensation for Ascend NPU, characterized in that, The method includes the following steps: Step 1: Obtain the input tensor and weight tensor to be performed inference calculation, and divide the input tensor and weight tensor into a set of input blocks and a set of weight blocks respectively according to a predetermined block division strategy, and establish the correspondence between the input blocks and the weight blocks to form a block pair set. Step 2: For the input block corresponding to the block index in the current layer and its corresponding weight block, calculate the statistical characteristics of the input block and / or its corresponding weight block, and determine the dynamic scaling factor based on the statistical characteristics; Step 3: Perform dynamic scaling processing on the input block and / or its corresponding weight block according to the dynamic scaling factor to obtain the input block and weight block after dynamic scaling processing; Step 4: On the matrix calculation unit of the Ascend NPU, perform matrix multiplication or convolution operations on the input block and weight block after the dynamic scaling process to obtain the basic output block; wherein, the dynamic scaling process causes one of the input block and / or the weight block to be scaled, while the other remains unchanged or is scaled inversely. Step 5: On the vector computing unit of Ascend NPU, error features are extracted based on the basic output block, and optionally combined with auxiliary features extracted from the input block corresponding to the basic output block to determine the error features; the error features are compared with a preset threshold to obtain a trigger flag; Step 6: If the trigger flag meets the preset compensation condition, then select K candidate components from the basic output block as Top-K components according to the preset importance index, and perform residual compensation operation on the Top-K components based on the compensation coefficient parameters determined in the offline calibration stage to obtain the compensated components. Step 7: Merge the compensation component with the basic output block to obtain the compensation output block; otherwise, directly use the basic output block as the compensation output block. Step 8: Write the compensated output block into the corresponding position of the current layer output tensor; if there are still unprocessed block indices in the current layer, update the block index and return to step 2 to process the next block pair; when all block pairs in the current layer have been processed, obtain the current layer output tensor, and use the current layer output tensor as the input for the next layer inference calculation, return to step 1 to process the next layer, until the inference calculation is completed, and the program ends.

2. The method according to claim 1, characterized in that, The statistical features in step 2 include at least one of the following: the maximum absolute value of the input block, the mean and variance of the input block, and the exponential range representation of the input block; the maximum absolute value of the weighted block corresponding to the input block, the mean and variance of the weighted block, and the exponential range representation of the weighted block.

3. The method according to claim 1, characterized in that, The method for determining the dynamic scaling factor in step 2 includes: calculating a candidate value set of scaling factors based on the statistical characteristics, and selecting from the candidate value set a scaling factor that causes the numerical range of the dynamically scaled input block and / or its corresponding weight block to fall within a preset FP16 safety range as the dynamic scaling factor; wherein, the candidate value set of scaling factors includes at least a scaling factor calculated by any of the following statistical measures: maximum absolute value, root mean square value, absolute value quantile, and a combined pruning threshold of the maximum absolute value and the quantile; and / or, the candidate value set of scaling factors may also include candidate scaling factors determined by FP16 exponential range constraints, compensation trigger thresholds, and Top-K compensation budget parameters; wherein, the statistical measures Characterizes the magnitude scale of the input block and / or its corresponding weighted block, and sets the target upper bound or target scale corresponding to the preset FP16 safety range as... Then the candidate scaling factor satisfies ,in To prevent division by zero constants, the candidate scaling factor can be further quantized into an integer power of two form. Where n is limited to a preset power range Integers within, where For the least power, The maximum power; the preset FP16 safety range is determined by the pre-calibrated exponent interval and the effective mantissa retention requirement; Furthermore, the statistic A can also be the fused effective amplitude A. fuse The fused effective amplitude is composed of the maximum absolute value m, the root mean square r, and the absolute value quantile threshold q. PK The result of fusion is: A fuse =(1-β)·A tail +β·r, where A tail =min(m,ρ eff · q PK ) ; p K =1-K / N; ρ eff =clip(ρ0+ρ1· (1-p K )+ρ2· I(E prev ≤θ),ρ min ,ρ max ), β∈[0,1] are the fusion weights; E prev I(·) is the feedback quantity of the error characteristics or triggering result of the previous block, and I(·) is the indicator function; the candidate scaling factor can be set according to A=A fuse Substitute the values ​​into the formula and quantize them into an integer power of two.

4. The method according to claim 1, characterized in that, The third step of performing dynamic scaling on the input block and / or its corresponding weight block includes: performing scaling on the input block by multiplying by the dynamic scaling factor, and performing scaling on the weight block by multiplying by the reciprocal of the dynamic scaling factor; or, performing scaling on the weight block by multiplying by the dynamic scaling factor, and performing scaling on the input block by multiplying by the reciprocal of the dynamic scaling factor; to ensure that the desired output scale of the matrix multiplication or convolution operation remains unchanged.

5. The method according to claim 1, characterized in that, The fifth step of extracting error features includes: calculating energy representation, dispersion representation, and / or saturation ratio representation for the basic output block on the vector calculation unit, and generating the error features based on a preset combination rule; wherein, the saturation ratio representation is used to represent the proportion of elements in the basic output block whose absolute value is greater than the saturation threshold; the preset combination rule includes at least normalizing and then weighting the sum of the representations, or performing vector concatenation and then linear mapping to obtain the error feature value.

6. The method according to claim 1, characterized in that, The selection criteria for the Top-K components in step 6 include at least one of the following or a combination thereof: (1) the absolute value of each component in the basic output block; (2) the compensation priority score of each component in the basic output block; (3) the saturation risk of each component in the basic output block; wherein, the compensation priority score includes at least: a score obtained by weighted fusion of the absolute value of the component and the error characteristics of the corresponding channel; the saturation risk includes at least: the ratio of the absolute value of the component to the saturation threshold, or a function value representing the distance between the component exponent and the FP16 exponent boundary; the index set and / or mask set corresponding to the Top-K components of the Top-K selection output are used to indicate the calculation position of the residual compensation operation.

7. The method according to claim 1, characterized in that, The residual compensation operation in step 6 includes: calculating the compensation component based on the current value of the Top-K component and the error characteristics according to a preset piecewise function; wherein the preset piecewise function includes at least two different linear mapping relationships, and the coefficients of each linear mapping relationship are determined by the compensation coefficient parameters determined in the offline calibration stage; the compensation coefficient parameters determined in the offline calibration stage include at least a piecewise threshold τ, a first linear mapping coefficient a1 and a first bias term b1, a second linear mapping coefficient a2 and a second bias term b2; the compensation coefficient parameters are obtained through offline calibration, which includes at least: obtaining the base output and reference output based on calibration data and calculating the residuals, and performing piecewise fitting on the residuals to determine the piecewise threshold and the coefficients and bias terms of each linear mapping; the compensation coefficient parameters determined in the offline calibration stage are stored layer by layer in the parameter storage device; the calculation of the compensation component is implemented on the vector calculation unit through element-wise multiplication and addition operations.

8. The method according to claim 1, characterized in that, Steps 4 and 5 are executed in a collaborative pipeline on the Ascend NPU, including: while the matrix computation unit performs matrix multiplication or convolution operations on the current input block and weight block, the vector computation unit performs error feature extraction and residual compensation operations on the basic output block of the previous block; and the input block, weight block and basic output block satisfy a predetermined alignment constraint so that the block dimension of the matrix multiplication or convolution operation matches the data layout of the vector compensation operation. Furthermore, the preset piecewise function can be explicitly expressed as: for any component in the Top-K index set, the residual estimate d is obtained. i (For example, from the reference value / high-precision estimate Y) ref,i With basic output Y 0,i The difference is obtained, or the residual mapping model obtained from offline calibration is used to represent Y. 0,i (Obtained by estimating the error characteristic E), and then calculating the compensation component Δ. i =sign(d i ) · g(|d i |), where g(|d|) = a1· |d| + b1 (|d| ≤ τ), g(|d|) = a2· |d| + b2 (|d| > τ); the compensation output is Y. i =Y 0,I +Δ i For the components that are not in the Top-K, let Δ i =0.

9. An FP16 accuracy compensation system for Ascend NPU, characterized in that, The system includes an inference execution device and a parameter storage device; wherein, the parameter storage device is used to store threshold parameters, Top-K filtering parameters, compensation coefficient parameters and scaling strategy parameters corresponding to each layer; The reasoning execution device includes: The block segmentation module is used to obtain the input tensor and weight tensor to be performed inference computation, and divide the input tensor and weight tensor into a set of input blocks and a set of weight blocks respectively according to a predetermined block segmentation strategy, and establish a correspondence between the input blocks and the weight blocks to form a block pair set. The feature calculation module is used to calculate the statistical features of the input block and / or its corresponding weight block for any block index in any network layer. A scaling factor determination module is used to determine a dynamic scaling factor based on the statistical characteristics; The dynamic scaling module is used to perform dynamic scaling processing on the input block and / or its corresponding weight block according to the dynamic scaling factor, so as to obtain the input block and weight block after dynamic scaling processing. The matrix calculation module is used to perform matrix multiplication or convolution operations on the dynamically scaled input blocks and weight blocks using the matrix calculation unit of the Ascend NPU to obtain the basic output blocks. An error feature extraction module is used to extract error features based on the basic output block using the vector computing unit of the Ascend NPU, and optionally combine the auxiliary features extracted from the input block corresponding to the basic output block to determine the error features; The trigger discrimination module is used to compare the error characteristics with a preset threshold to obtain a trigger flag; The Top-K filtering module is used to filter K candidate components from the basic output block as Top-K components according to a preset importance index when the trigger flag meets the preset compensation conditions. The residual compensation module is used to perform residual compensation calculations on the Top-K components based on the compensation coefficient parameters determined in the offline calibration stage to obtain the compensated components; The fusion module is used to fuse the compensation component with the basic output block to obtain the compensation output block; The output and scheduling module is used to write the compensated output blocks into the corresponding positions of the current layer output tensor, and to schedule each module to execute in a loop when there are unprocessed block indices in the current layer; when all block pairs in the current layer have been processed, the current layer output tensor is output and used as the input for the next layer inference calculation, until the inference calculation is completed.

Images