Artificial intelligence processor tensor operator precision adaptive regulation method

Abstract

The application provides an artificial intelligence processor tensor operator precision self-adaptive regulation method, including the following steps: receiving the instruction code of the tensor operator, identifying the operation on the tensor operator as a tensor operation with multi-level rounding error; obtaining the current working mode of the artificial intelligence processor, the current working mode including reasoning and training; selecting a bypass operation path of the tensor operation according to the instruction code of the tensor operator, the bypass operation path being parallel to a standard operation path of the tensor operation, and all intermediate results in the bypass operation path being abandoned from any intermediate rounding and truncation; performing cascade operation in the bypass operation path, the mantissa of all intermediate results being additionally reserved with multiple guard bits to maintain precision; and after detecting the cascade operation end signal of the tensor operator, performing self-adaptive segmented offset rounding on the cascade operation result.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence processors, and specifically relates to a method for adaptive control of the precision of tensor operators in artificial intelligence processors. Background Technology

[0002] With the exponential growth of parameters in Large Language Models (LLMs), model training and inference place extremely high demands on computational accuracy and hardware energy efficiency. The normalization calculations of the Floating Point Processing Unit (FPU) in AI processors involve multiple levels of iterative operations such as mean, variance, reciprocal square root, and scaling. These computational chains are all strictly serial, dependent data flows. Standard FPUs use a round-to-nearest-even rounding rule for rounding. However, the normalized mantissa m is not uniformly distributed. This non-uniform distribution causes the "round-to-nearest-even rounding" rule to generate a unidirectional, cumulative systematic bias in key operations such as mean, variance, and reciprocal square root over thousands of iterations. Each rounding error propagates to the next step, resulting in accumulated rounding errors, which manifest as a continuous decay of the gradient signal (gradient vanishing) during backpropagation. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a method for adaptively adjusting the precision of tensor operators in artificial intelligence processors, characterized by the following steps:

[0004] Receive the instruction code of the tensor operator and identify the operation on the tensor operator as a tensor operation with multi-level rounding error;

[0005] Obtain the current operating mode of the artificial intelligence processor, the current operating mode including inference and training;

[0006] The bypass operation path of the tensor operation is selected according to the instruction code of the tensor operator. The bypass operation path runs in parallel with the standard operation path of the tensor operation. In the bypass operation path, all intermediate results are discarded and any intermediate rounding and truncation are performed.

[0007] The cascaded operations in the bypass operation path are executed, and the mantissa of all intermediate results is reserved with multiple extra guard bits to maintain accuracy;

[0008] After detecting the end signal of the cascade operation of the tensor operator, the cascade operation result of the tensor operator is adaptively segmented and offset rounded according to the current working mode and the protection bit;

[0009] Write the final rounding result to the target register or instruction queue, reset the state machine during the cascade operation of the tensor operator, and clear the temporary cache.

[0010] In one embodiment of the aforementioned method, adaptive piecewise offset rounding of the cascaded operation result of the tensor operator based on the current operating mode and the protection bit further includes:

[0011] The segmentation of the mantissa of the cascaded operation result is determined according to the type of the tensor operator, and the protection bit is dynamically offset and rounded according to the segmentation of the mantissa and the current working mode, corresponding to the current working mode and the segmentation of the mantissa.

[0012] In one embodiment of the aforementioned method, the dynamic offset rounding of the protection bit based on the segmentation of the mantissa and the current operating mode, corresponding to the current operating mode and the segmentation of the mantissa, further includes:

[0013]

[0014] in, The function definition for dynamic offset rounding, For protection purposes, For the last digit, For the mantissa width, For tensor operators, As for the current work mode, The adaptive offset adjustment function is based on the mantissa, tensor operator type, and current working mode. When the current working mode is training mode, the adaptive offset adjustment function is an alternating positive and negative offset function based on the mantissa. When the current working mode is inference mode, the adaptive offset adjustment function is a static offset function based on the mantissa.

[0015] In one embodiment of the aforementioned method, the adaptive offset adjustment function is:

[0016]

[0017] Wherein, the alternation offset coefficient k= , The scaling factor corresponds to the segmented interval, and n is the bit width adjustment variable corresponding to the tensor operator type.

[0018] In one embodiment of the aforementioned method, the current operating mode is set via a status register.

[0019] The present invention also proposes an artificial intelligence processor, comprising:

[0020] An instruction decoder is used to receive instruction codes of tensor operators and identify operations on the tensor operators as tensor operations with multi-level rounding errors.

[0021] A mode switching controller is used to acquire the current operating mode of the artificial intelligence processor, the current operating mode including inference and training; and,

[0022] The bypass operation path of the tensor operation is selected according to the instruction code of the tensor operator. The bypass operation path runs in parallel with the standard operation path of the tensor operation. In the bypass operation path, all intermediate results are discarded and any intermediate rounding and truncation are performed.

[0023] The iterative precision preservation module is used to perform cascaded operations in the bypass operation path, and the mantissa of all intermediate results is additionally protected by multiple guard bits to maintain precision.

[0024] The adaptive segmented offset rounding module is used to detect the end signal of the cascade operation of the tensor operator, and then perform adaptive segmented offset rounding on the cascade operation result of the tensor operator according to the current working mode and the protection bit; and write the final rounding result into the target register or instruction queue, reset the state machine in the cascade operation process of the tensor operator, and clear the temporary cache.

[0025] In one embodiment of the aforementioned artificial intelligence processor, the adaptive piecewise offset rounding module performs adaptive piecewise offset rounding on the cascaded operation result of the tensor operator based on the current operating mode and the guard bit, further comprising:

[0026] The segmentation of the mantissa of the cascaded operation result is determined according to the type of the tensor operator, and the protection bit is dynamically offset and rounded according to the segmentation of the mantissa and the current working mode, corresponding to the current working mode and the segmentation of the mantissa.

[0027] In one embodiment of the aforementioned artificial intelligence processor, the dynamic offset rounding of the protection bit based on the segmentation of the mantissa and the current operating mode, corresponding to the current operating mode and the segmentation of the mantissa, further includes:

[0028]

[0029] in, The function definition for dynamic offset rounding, For protection purposes, For the last digit, For the mantissa width, For tensor operators, As for the current work mode, The adaptive offset adjustment function is based on the mantissa, tensor operator type, and current working mode. When the current working mode is training mode, the adaptive offset adjustment function is an alternating positive and negative offset function based on the mantissa. When the current working mode is inference mode, the adaptive offset adjustment function is a static offset function based on the mantissa.

[0030] In one embodiment of the aforementioned artificial intelligence processor, the adaptive offset adjustment function is:

[0031]

[0032] Wherein, the alternation offset coefficient k= , The scaling factor corresponds to the segmented interval, and n is the bit width adjustment variable corresponding to the tensor operator type.

[0033] In one embodiment of the aforementioned artificial intelligence processor, the mode switching controller switches the current operating mode via a status register.

[0034] Compared to existing technologies, the artificial intelligence processor tensor operator precision adaptive control method and artificial intelligence processor disclosed in this invention select the bypass operation path of tensor operation according to the instruction code of the tensor operator; the bypass operation path runs in parallel with the standard operation path of tensor operation, and no intermediate results in the bypass operation path are subjected to any intermediate rounding or truncation; the error propagation path of "step-by-step rounding / truncation" is blocked at the hardware level, providing a core physical guarantee for maintaining iterative accuracy.

[0035] When performing cascaded operations in the bypass operation path, multiple guard bits are reserved for the mantissa of all intermediate results to maintain accuracy; this ensures that significant digits are not lost due to alignment shifts during multiple multiplication-addition operations and nonlinear iterations, thereby blocking the propagation path of errors between stages.

[0036] After detecting the end signal of the cascade operation of tensor operators, adaptive segmented offset rounding is performed on the cascade operation result of tensor operators according to the current working mode and protection bit. By delaying the rounding decision point to the end of the computation chain, adaptive segmented offset rounding is performed on the final result that has accumulated full-precision information to maintain the high precision of the calculation result, while taking into account the training gradient fidelity and inference energy efficiency.

[0037] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0038] Figure 1A flowchart illustrating an embodiment of the present invention shows a method for adaptive adjustment of the precision of tensor operators in an artificial intelligence processor.

[0039] Figure 2 A block diagram illustrating an artificial intelligence processor according to an embodiment of the present invention is shown.

[0040] Figure 3 A block diagram illustrating the normalization operation module of an artificial intelligence processor in one embodiment of the present invention is shown.

[0041] In the attached figures, the following labels are used:

[0042] 1…Instruction decoder

[0043] 10…Mode Switching Controller

[0044] 20… Iterative Precision Preservation Module

[0045] 30…Adaptive Piecewise Offset Rounding Module

[0046] 40…computing cores

[0047] 100… Artificial Intelligence Processor

[0048] S100-S600…Steps Detailed Implementation

[0049] The following specific embodiments, in conjunction with the accompanying drawings, illustrate the implementation methods disclosed in this invention. Those skilled in the art can understand the advantages and effects of this invention from the content disclosed in this specification. However, the following disclosure is not intended to limit the scope of protection of this invention. Without departing from the spirit of the invention, those skilled in the art can implement this invention with other different embodiments based on different viewpoints and applications.

[0050] For clarity, the figures shown in this invention are simplified schematic diagrams illustrating the basic structure of the invention. Therefore, the structures shown in the figures are not drawn to scale according to the actual shape and size of the implementation. For example, the dimensions of certain components have been enlarged for ease of explanation.

[0051] Furthermore, it should be understood that when a component such as a layer, film, region, or substrate is referred to as being "on" or "connected" to another component, it may be directly on or connected to the other component, or an intermediate component may also be present. Conversely, when a component is referred to as being "directly on" or "directly connected" to another component, no intermediate component exists. As used herein, "connection" can refer to physical and / or electrical connections. Moreover, "electrical connection" or "coupling" can refer to the presence of other components between the two components.

[0052] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this invention pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as they have in the context of the relevant technology and this invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0053] Furthermore, it should be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various components, parts, regions, layers, and / or portions, these components, parts, regions, and / or portions should not be limited by these terms. These terms are used only to distinguish one component, part, region, layer, or portion from another. Therefore, the “first component,” “part,” “region,” “layer,” or “part” discussed below may be referred to as a second component, part, region, layer, or portion without departing from the teachings of this document.

[0054] It should be understood that references to "an embodiment," "embodiment," "example embodiment," etc., in the specification refer to the described embodiment, indicating that it may include specific features, structures, or characteristics, but does not necessarily include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0055] The specification and subsequent claims use certain terms to refer to specific modules, components, or parts. Those skilled in the art will understand that users or manufacturers may use different names or terms to refer to the same module, component, or part. This specification and subsequent claims do not distinguish modules, components, or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Furthermore, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections via other means.

[0056] Furthermore, in the following description and claims, numerous terms will be referenced, which should be defined as having the following meanings. The singular forms “a” and “the” include plural referents, unless the context clearly specifies otherwise. “Optional” or “optionally” indicates that an event or situation subsequently described may or may not occur, and the description includes both the scenario where the event occurs and the scenario where the event does not occur.

[0057] Please see Figure 1 , Figure 1 A flowchart illustrating an embodiment of the present invention provides a method for adaptively adjusting the precision of tensor operators in an artificial intelligence processor. To construct a hardware-native, precision-controllable, and mode-adaptive floating-point arithmetic microarchitecture, the present invention proposes a method for adaptively adjusting the precision of tensor operators in an artificial intelligence processor, comprising the following steps:

[0058] S100: Receives instruction codes for tensor operators and identifies operations on tensor operators as tensor operations with multi-level rounding errors;

[0059] In one embodiment, an instruction code is received by an instruction decoder, and it is detected whether it matches a predefined normalization-specific microcode to accurately distinguish between normalization operations and general calculations. This allows the tensor operator precision adaptive control method of the present invention to target normalization tasks (or other tensor operations with multi-level rounding errors as needed), avoid interference from other customized logic on non-normalization tasks, and provide a unified trigger signal for subsequent calculation tasks.

[0060] S200: Obtain the current operating mode of the artificial intelligence processor, which includes inference and training;

[0061] After identifying it as a normalization operation, the internal control status register is immediately read to lock the current operating mode.

[0062] In multi-task scenarios, this approach balances gradient fidelity during training with energy efficiency during inference, taking into account the characteristics of both training and inference modes. The same arithmetic logic unit and storage system can seamlessly switch between training and inference modes without requiring circuit modifications or model weight recalibration for specific modes. This integrated training and inference feature eliminates the format conversion overhead common in heterogeneous deployments, enabling data centers to dynamically reuse computing resources based on business loads, significantly reducing the total cost of ownership and operational complexity of the infrastructure.

[0063] S300: Selects a bypass operation path for tensor operations based on the instruction code of the tensor operator. The bypass operation path runs in parallel with the standard operation path for tensor operations. In this bypass operation path, all intermediate results are discarded and any intermediate rounding and truncation are performed.

[0064] Therefore, blocking the error propagation path of "rounding / truncation" at the hardware level provides a core physical guarantee for maintaining iterative accuracy.

[0065] S400: Performs cascaded operations in the bypass operation path, and retains multiple extra guard bits in the mantissa of all intermediate results to maintain precision;

[0066] This invention adds 4 to 8 extra guard bits to the mantissa portion of all intermediate results while maintaining the full precision of the exponent portion. This ensures that significant digits are not lost due to alignment shifts during multiple multiplication-addition operations and nonlinear iterations, thereby blocking the propagation path of errors between stages.

[0067] S500: After detecting the end signal of the cascade operation of tensor operators, adaptive segmented offset rounding is performed on the cascade operation result of tensor operators according to the current working mode and protection bits.

[0068] By delaying the rounding decision point to the end of the computation chain, adaptive piecewise offset rounding is performed on the final result that has accumulated full-precision information, maintaining the high precision of the computation result while taking into account both training gradient fidelity and inference energy efficiency.

[0069] In one embodiment, after step S500, step S600 is executed: the final rounding result is written to the target register or instruction queue, the state machine in this tensor operator cascade operation is reset, and the temporary cache is cleared.

[0070] In this way, the external interface can be guaranteed to be compatible with the IEEE-754 standard and seamlessly integrated with the existing AI software stack; after the final rounding result is written to the target register or instruction queue, the internal state is cleared and the system waits for the next calculation process to be triggered, thus avoiding precision pollution to subsequent calculations and realizing safe reuse of hardware resources.

[0071] In steps S100 to S600, the tensor operators in this invention are described using the normalization operators LayerNorm and RMSNorm as examples. Tensor operators can also be nonlinear activation operators (Softmax, Sigmoid) or convolution operators. The tensor operations of the above types of tensor operators all have tensor operations with multi-level rounding errors. The types of tensor operators in this invention are not limited to these.

[0072] In steps S300 and S400, in the prior art, to match the fixed-width register file and reduce power consumption, the output of each stage of the arithmetic logic unit (ALU) / floating-point processing unit (FPU) is immediately rounded and truncated to align the result to a standard format (such as FP16 or BF16). However, the "step-by-step truncation" strategy introduces a serious error accumulation problem. The quantization error of the previous stage is used as the input of the next stage, amplified by a nonlinear function (such as the reciprocal of the square root), causing the final normalized output to deviate from the theoretical value, resulting in significant accuracy loss in deep networks.

[0073] Therefore, in this invention, in step S300, when the instruction decoder recognizes the microcode sequence dedicated to normalization, a path switching control signal is activated. This signal acts on the data path selector to determine the bypass operation path for tensor operations. This bypass operation path runs parallel to the standard operation path for tensor operations, forcing subsequent adder trees, multipliers, and approximate function calculation units (e.g., calculating the reciprocal square root rSqrt) to bypass the standard rounding logic and enter a "high-precision pass-through mode." That is, the output of each level of arithmetic logic unit (ALU) / floating-point processing unit (FPU) in the bypass operation path will not be rounded or truncated. In the normalization cascade operation, the LayerNorm and RMSnorm operators, as the core components of the normalization layer of the Transformer structure, involve multi-level iterative operations such as mean, variance, reciprocal square root (rSqrt), and scaling.

[0074] Both LayerNorm and RMSNorm, at the underlying hardware level, employ strictly serial, dependent data flows (each subsequent stage must wait for the previous stage to complete).

[0075] RMSNorm path: Sum of squares (S) → Mean (S / n) → Reciprocal square root (rSqrt) → Scaling multiplication.

[0076] LayerNorm path: Calculate mean → Calculate sum of squared deviations → Calculate variance → Calculate reciprocal square root (rSqrt) → Scaling and offset.

[0077] The standard FPU uses a round-to-nearest-even rounding rule. The normalized mantissa m is not uniformly distributed, but rather largely concentrated in the [0, 0.5) interval. This non-uniform distribution causes a unidirectional accumulation of systematic bias in key operations such as the mean, variance, and reciprocal square root over thousands of iterations. Each rounding error propagates to the next, resulting in accumulated rounding errors, which manifest as a continuous decay of the gradient signal (gradient loss) during backpropagation. Therefore, in step S400, during the cascaded calculation process, all intermediate results are no longer truncated to the standard bit width, but instead retain an extended bit width. In one embodiment, an additional 4-8 guard bits are retained for the mantissa of the intermediate results, thus maintaining full precision and ensuring that significant digits from multiple multiplications and nonlinear iterations are not lost due to alignment shifts, thereby blocking the propagation path of errors between stages.

[0078] In step S500, after the signal indicating the end of the cascade operation of tensor operators is detected, a one-time high-precision rounding operation is performed on the final result (the result of the cascade operation of tensor operators) that has accumulated full-precision information. In other words, this invention postpones the rounding decision point to the end of the computation chain, and performs high-precision rounding only once at the end of the normalization cascade operation, completely avoiding the amplification of errors across stages, thereby blocking the propagation path of errors between stages.

[0079] In one embodiment of the above method, the step of adaptively rounding the cascaded operation result of the tensor operator according to the current working mode and the protection bit in step S500 further includes:

[0080] The segmentation of the mantissa of the cascaded operation result is determined according to the type of tensor operator, and the protection bits are dynamically offset and rounded according to the segmentation of the mantissa and the current working mode.

[0081] Specifically, let the mantissa after floating-point arithmetic be... The current floating-point format has a mantissa width of 1. (BF16) In FP16 The standard rounding threshold is... .

[0082] When the tensor operator type is LayerNorm, Divided into three sections, among which For small value range, For the medium value range, For large numerical regions. When the tensor operator type is RMSNorm, Divided into two sections, among which It is a low-energy region. This is a high-energy region.

[0083] For the LayerNorm operator, since The numerical distribution is relatively scattered, and after standardization, the data shows a relatively symmetrical and concentrated distribution near 0. The range of values ​​for the mantissa m is divided into three segments to adapt to the statistical characteristics of the standardized data at different magnitudes, so as to better maintain gradient fidelity during the final rounding.

[0084] For the RMSNorm operator, the data distribution after computation typically exhibits a more unidirectional or energy-concentrated characteristic. Therefore, dividing the data into two segments based on energy level satisfies the accuracy requirements for inference or training and adapts to the statistical characteristics of the standardized data across different energy segments. Similarly, when segmenting the mantissa of the cascaded operation results of other types of tensor operators, segmentation can also be based on the statistical characteristics of different segments of that tensor operator type.

[0085] In one embodiment of the above method, the step of performing dynamic offset rounding on the protection bit according to the segmentation of the mantissa and the current operating mode, corresponding to the current operating mode and the segmentation of the mantissa, further includes:

[0086]

[0087] in, Function definition for dynamic offset rounding;

[0088] To protect the bit, the mantissa addition will exceed the current format. The lower digit is the basis for rounding decisions;

[0089] The mantissa is the decimal part of a normalized floating-point number, reflecting the precision of the value.

[0090] The mantissa width is the hardware-defined floating-point format precision.

[0091] It is a tensor operator type, a normalization operator type, and in this embodiment, it is an enumeration variable. This is used to adapt to different segmentation strategies;

[0092] The current working mode, including training mode and inference mode, is an enumerated variable in this embodiment. ;

[0093] The adaptive offset adjustment function is based on the mantissa, tensor operator type, and current working mode. When the current working mode is training mode, the adaptive offset adjustment function is an alternating positive and negative offset function based on the mantissa. When the current working mode is inference mode, the adaptive offset adjustment function is a static offset function based on the mantissa.

[0094] The standardized mantissa of floating-point operations is The current floating-point format has a mantissa width of 1. (BF16) In FP16 The standard rounding threshold is... .

[0095] In one embodiment of the above method, the adaptive offset adjustment function is:

[0096]

[0097] Wherein, the alternation offset coefficient k= , i is the mapping coefficient, symbol " "For multiplication, the symbol " "To round down" The scaling factor corresponds to the segmented interval, and n is the bit width adjustment variable corresponding to the tensor operator type.

[0098] In this embodiment, when the tensor operator type is LayerNorm, and the bit width adjustment variable n is 2, For small value range, For the medium value range, This is a region with large numerical values.

[0099] Alternating offset coefficient k= , ,because The numerical distribution is relatively scattered, so a three-segment static offset is adopted, and a small numerical range is assigned... The contraction correction maintains a base of 1.0 in the medium value range and an expansion correction of 1.2 in the large value range to adapt to the characteristics of its dynamic range.

[0100] Alternating offset coefficient k= The value range of k is {0, 1, 2, 3}. The four k values ​​correspond exactly to the boundary determination of the three segmented intervals ([0, 0.25) corresponds to k=0, [0.25, 0.75) corresponds to k=1, 2, and [0.75, 1.0) corresponds to k=3, realizing alternating positive and negative offsets.

[0101] When the pattern This is the lossless mode, based on the last digit. In the segmented interval, positive or negative offsets are applied alternately. By utilizing the statistical zero mean principle, the systematic bias of standard rounding is offset, gradient loss is eliminated, and lossless training is achieved.

[0102] When the pattern For lossy mode, the numerical distribution prior of the normalization operator is used to apply static and deterministic offset correction to the alternating offset coefficients k corresponding to different mantissas, so as to maximize the approximation of lossless accuracy under the premise of controllable hardware overhead.

[0103] When the tensor operator type is RMSNorm, the bit width adjustment variable n is 3. It is divided into two parts, among which It is a low-energy region. It is a high-energy region. ,

[0104] ,because The numerical distribution is relatively concentrated, so a two-stage static migration is adopted, and the low-energy region is given... Slight contraction, high energy region endows A slight expansion allows for more precise quantitative control.

[0105] In one embodiment of the above method, the current operating mode in step S200 is set by a status register.

[0106] The controller can achieve one-click hardware-level switching through the configuration status register (CSR), without the need for software intervention in data format conversion.

[0107] Table 1

[0108]

[0109] This invention achieves a unified hardware architecture through the Controller Configuration Status Register (CSR), enabling hardware-level single-cycle mode switching without software recompilation. This provides a foundation for dynamic reuse of training and inference computing power. The same set of arithmetic logic units and storage system can seamlessly switch between training and inference modes without modifying circuits or recalibrating model weights for specific modes. The "integrated training and inference" feature eliminates the format conversion overhead common in heterogeneous deployments, allowing data centers to dynamically reuse computing resources based on business load, greatly reducing the total cost of ownership and operational complexity of infrastructure.

[0110] In one embodiment, the mantissa is normalized using LayerNorm. Taking offset calculation and rounding correction as an example:

[0111] Normalization operator types (tensor operator types):

[0112] Current working mode:

[0113] Mantissa (the number of significant decimal places before normalization and scaling):

[0114] Precision bit width (The actual number of significant digits in the mantissa of BF16 is approximately 7-8 digits; a conservative value is used here.) (For example)

[0115] Standard rounding threshold: ,

[0116] First, determine the interval to which the last digit m belongs.

[0117] It belongs to the median value range. ;

[0118] Calculate the alternating offset coefficient ;

[0119] Calculate the offsets in training and inference modes separately. :

[0120] when :

[0121] ,

[0122] This offset will be added to the standard rounding result, using the statistical zero mean principle to offset systematic bias in a large number of samples, thus achieving "lossless training".

[0123] when :

[0124] ,

[0125] This offset is a static preset value that is adaptively adjusted based on the characteristics of the normalization operator (the numerical region in LayerNorm). Within the limits allowed by the hardware, the rounding direction is finely adjusted to approximate lossless accuracy.

[0126] Apply offset for rounding correction:

[0127] Assuming the value before standard rounding is Apply the above Make corrections:

[0128] Training mode:

[0129]

[0130] If the value needs to be rounded to the nearest integer... (i.e., multiples of 0.00048828125), then:

[0131] ,

[0132] Reasoning Mode:

[0133] ,

[0134] .

[0135] As can be seen, the offset is larger in inference mode, but through static configuration... It can still approach the optimal rounding point without sacrificing speed.

[0136] Table 2

[0137]

[0138] The calculation results of the tensor operator precision adaptive control method of this invention in training mode and inference mode are shown in Table 1. The deviation between the training mode and the theoretical true value of this invention is lower than that of the standard rounding mode, and it has higher computational accuracy. Although the absolute error of the inference mode is slightly larger, its advantages are that it does not require dynamic statistics, does not require additional control logic, has lower power consumption, and the average error in batch data approaches zero (because...). (Static configuration based on intervals, conforming to the prior distribution).

[0139] like At this point, the interval is in the range of large values. ,

[0140] In training mode:

[0141] In reasoning mode: 0.0005859375

[0142] It is evident that a negative offset is applied during training and a positive expansion correction is applied during inference in the large numerical region to suit its high dynamic range characteristics.

[0143] Please see Figure 2 and Figure 3 , Figure 2 A block diagram illustrating an artificial intelligence processor according to an embodiment of the present invention is shown. Figure 3 This invention illustrates a block diagram showing the normalization operation of an artificial intelligence processor in one embodiment of the invention. The invention also proposes an artificial intelligence processor 100, comprising:

[0144] Instruction decoder 1 is used to receive instruction codes of tensor operators and identify operations on tensor operators as tensor operations with multi-level rounding errors;

[0145] The mode switching controller 10 is used to acquire the current operating mode of the artificial intelligence processor, which includes inference and training. Upon identifying an operation on a tensor operator as a tensor operation with multi-level rounding errors (e.g., normalization), it immediately reads the internal control status register and locks the current operating mode.

[0146] The bypass operation path of tensor operation is selected according to the instruction code of the tensor operator. The bypass operation path runs in parallel with the standard operation path of tensor operation. In the bypass operation path, all intermediate results are discarded and no intermediate rounding or truncation is performed. The iteration precision preservation module 20 is used to execute the cascade operation in the bypass operation path. The mantissa of all intermediate results is additionally preserved with multiple guard bits to maintain precision.

[0147] The adaptive segmented offset rounding module 30 is used to perform adaptive segmented offset rounding on the cascaded operation result of the tensor operator according to the current working mode and protection bits after detecting the end signal of the cascaded operation of the tensor operator; and to write the final rounding result into the target register or instruction queue, reset the state machine in the cascaded operation process of the tensor operator, and clear the temporary cache.

[0148] like Figure 2 and Figure 3 As shown, the mode switching controller 10, based on the tensor operator operation identified by the instruction decoder as a tensor operation with multi-level rounding errors, configures the current working mode (inference or training) to the adaptive segmented offset rounding module 30. Simultaneously, the mode switching controller 10 configures the iterative precision preservation module 20 to the precision adaptive rounding mode. The iterative precision preservation module 20 transmits a path switching control signal to the computation core 40. This signal acts on the data path selector, determining the bypass operation path for tensor operations. This bypass operation path runs parallel to the standard operation path for tensor operations, forcing subsequent adder trees, multipliers, and approximate function calculation units (e.g., calculating the reciprocal square root rSqrt) to bypass the standard rounding logic and enter a "high-precision pass-through mode." That is, the output of each stage of the pipeline in the bypass operation path's arithmetic logic unit (ALU) / floating-point processing unit (FPU) will not be rounded or truncated, thus blocking the error propagation path of "step-by-step rounding / truncation" at the hardware level. (It should be noted that...) Figure 3Although the basic computation units in the high-precision pass-through mode pipeline and the standard mode pipeline are shown separately for ease of understanding (the basic computation units are the same set of computation units), tensor operations in the standard rounding mode are unaffected. The same set of arithmetic logic units and storage system can seamlessly switch between training and inference modes without modifying circuits or recalibrating model weights for specific modes, facilitating flexible deployment and resource reuse. In the cascaded operations of the bypass computation path, multiple extra guard bits are retained in the mantissa of all intermediate results to maintain accuracy. This invention adds 4 to 8 extra guard bits to the mantissa of all intermediate results. The mantissa of all intermediate results (S, S / n, rSqrt iterative initial values) retains p+4 to p+8 bits (guard bits), while maintaining full precision of the exponent part. This ensures that significant digits are not lost due to alignment shifts during multiple multiplication-addition operations and nonlinear iterations, thereby blocking the propagation path of errors between stages. After detecting the end signal of the cascade operation of tensor operators, the adaptive segmented offset rounding module 30 performs adaptive segmented offset rounding according to the current working mode and the cascade operation result. By delaying the rounding decision point to the end of the computation chain, adaptive segmented offset rounding is performed on the final result that has accumulated full-precision information, maintaining the high precision of the calculation result while taking into account the fidelity of training gradients and inference efficiency. After the operation is completed, the final rounding result is written to the target register or instruction queue, the state machine in this tensor operator cascade operation is reset, and the temporary cache is cleared. In this way, the external interface is compatible with the IEEE-754 standard and can be seamlessly connected to the existing AI software stack. After writing the final rounding result to the target register or instruction queue, the internal state is cleared and the system waits for the next operation to be triggered, avoiding precision pollution to subsequent operations and realizing safe reuse of hardware resources. In one embodiment of the aforementioned artificial intelligence processor 100, the adaptive segmented offset rounding module 30 further includes adaptive segmented offset rounding of the cascade operation result of tensor operators according to the current working mode and protection bits, including:

[0149] The segmentation of the mantissa of the cascaded operation result is determined according to the type of tensor operator, and the protection bits are dynamically offset and rounded according to the segmentation of the mantissa and the current working mode.

[0150] In one embodiment of the aforementioned artificial intelligence processor 100, the dynamic offset rounding of the protection bit based on the segmentation of the mantissa and the current operating mode, corresponding to the current operating mode and the segmentation of the mantissa, further includes:

[0151]

[0152] in, The function definition for dynamic offset rounding, For protection purposes, For the last digit, For the mantissa width, For tensor operators, As for the current work mode, The adaptive offset adjustment function is based on the mantissa, tensor operator type, and current working mode. When the current working mode is training mode, the adaptive offset adjustment function is an alternating positive and negative offset function based on the mantissa. When the current working mode is inference mode, the adaptive offset adjustment function is a static offset function based on the mantissa.

[0153] In one embodiment of the aforementioned artificial intelligence processor 100, the adaptive offset adjustment function is:

[0154]

[0155] Wherein, the alternation offset coefficient k= , The scaling factor corresponds to the segmented interval, and n is the bit width adjustment variable corresponding to the tensor operator type.

[0156] In one embodiment of the aforementioned artificial intelligence processor 100, the mode switching controller 10 switches the current working mode by setting the status register.

[0157] Compared to existing technologies, the beneficial effects of the artificial intelligence processor tensor operator precision adaptive control method and the artificial intelligence processor disclosed in this invention include:

[0158] The bypass operation path of tensor operation is selected according to the instruction code of the tensor operator; the bypass operation path runs in parallel with the standard operation path of tensor operation, and no intermediate results in the bypass operation path are subject to any intermediate rounding or truncation; the error propagation path of "step-by-step rounding / truncation" is blocked at the hardware level, providing core physical guarantee for maintaining the accuracy of iteration.

[0159] When performing cascaded operations in the bypass operation path, multiple guard bits are reserved for the mantissa of all intermediate results to maintain accuracy; this ensures that significant digits are not lost due to alignment shifts during multiple multiplication-addition operations and nonlinear iterations, thereby blocking the propagation path of errors between stages.

[0160] After detecting the end signal of the cascade operation of tensor operators, adaptive segmented offset rounding is performed on the cascade operation result of tensor operators according to the current working mode and protection bit. By delaying the rounding decision point to the end of the computation chain, adaptive segmented offset rounding is performed on the final result that has accumulated full-precision information to maintain the high precision of the calculation result, while taking into account the training gradient fidelity and inference energy efficiency.

[0161] In inference mode, the static adaptive rounding strategy effectively compresses the dynamic range of activation values ​​through normalization-aware coefficient configuration. By avoiding frequent switching of runtime dynamic statistical circuits, the hardware control logic is simplified, reducing the need for on-chip memory protection bits and alleviating memory bandwidth pressure. The power consumption of the normalization calculation unit is reduced by approximately 15%-20% compared to traditional dynamic rounding architectures.

[0162] This invention achieves a unified hardware architecture through a mode switching controller. The same set of arithmetic logic units and storage system can seamlessly switch between training and inference modes without modifying circuits or recalibrating model weights for specific modes. This "training-integrated" feature eliminates the format conversion overhead common in heterogeneous deployments, enabling data centers to dynamically reuse computing resources based on business load, greatly reducing the total cost of ownership (TCO) and operational complexity of the infrastructure.

[0163] The above-disclosed content is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of the patent application of the present invention. Therefore, any equivalent technical changes made based on the contents of the present invention specification and drawings fall within the scope of the patent application of the present invention.

Claims

1. A method for adaptively adjusting the precision of tensor operators in an artificial intelligence processor, characterized in that, Including the following steps: Receive the instruction code of the tensor operator and identify the operation on the tensor operator as a tensor operation with multi-level rounding error; Obtain the current operating mode of the artificial intelligence processor, the current operating mode including inference and training; The bypass operation path of the tensor operation is selected according to the instruction code of the tensor operator. The bypass operation path runs in parallel with the standard operation path of the tensor operation. In the bypass operation path, all intermediate results are discarded and any intermediate rounding and truncation are performed. The cascaded operations in the bypass operation path are executed, and the mantissa of all intermediate results is reserved with multiple extra guard bits to maintain accuracy; After detecting the end signal of the cascade operation of the tensor operator, the cascade operation result of the tensor operator is adaptively segmented and offset rounded according to the current working mode and the protection bit.

2. The method according to claim 1, characterized in that, The adaptive piecewise offset rounding of the cascaded operation result of the tensor operator based on the current working mode and the protection bit further includes: The segmentation of the mantissa of the cascaded operation result is determined according to the type of the tensor operator, and the protection bit is dynamically offset and rounded according to the segmentation of the mantissa and the current working mode, corresponding to the current working mode and the segmentation of the mantissa.

3. The method according to claim 1, characterized in that, The dynamic offset rounding of the protection bit based on the segmentation of the mantissa and the current operating mode, corresponding to the current operating mode and the segmentation of the mantissa, further includes: in, The function definition for dynamic offset rounding, For protection purposes, For the last digit, For the mantissa width, For tensor operators, As for the current work mode, The adaptive offset adjustment function is based on the mantissa, tensor operator type, and current working mode. When the current working mode is training mode, the adaptive offset adjustment function is an alternating positive and negative offset function based on the mantissa. When the current working mode is inference mode, the adaptive offset adjustment function is a static offset function based on the mantissa.

4. The method according to claim 3, characterized in that, The adaptive offset adjustment function is: Wherein, the alternation offset coefficient k= , The scaling factor corresponds to the segmented interval, and n is the bit width adjustment variable corresponding to the tensor operator type.

5. The method according to any one of claims 1 to 4, characterized in that, The current operating mode is set via the status register.

6. An artificial intelligence processor, characterized in that, include: An instruction decoder is used to receive instruction codes of tensor operators and identify operations on the tensor operators as tensor operations with multi-level rounding errors. A mode switching controller is used to obtain the current working mode of the artificial intelligence processor, the current working mode including inference and training; and to select a bypass operation path of the tensor operation according to the instruction code of the tensor operator, the bypass operation path being parallel to the standard operation path of the tensor operation, wherein all intermediate results are discarded and any intermediate rounding and truncation are performed in the bypass operation path. The iterative precision preservation module is used to perform cascaded operations in the bypass operation path, and the mantissa of all intermediate results is additionally protected by multiple guard bits to maintain precision. An adaptive segmented offset rounding module is used to perform adaptive segmented offset rounding on the cascaded operation result of the tensor operator according to the current working mode and the protection bit after detecting the end signal of the cascaded operation of the tensor operator. Additionally, the final rounding result is written to the target register or instruction queue, the state machine is reset during the cascade operation of the tensor operator, and the temporary cache is cleared.

7. The artificial intelligence processor according to claim 6, characterized in that, The adaptive piecewise offset rounding module performs adaptive piecewise offset rounding on the cascaded operation result of the tensor operator based on the current working mode and the protection bit, further including: The segmentation of the mantissa of the cascaded operation result is determined according to the type of the tensor operator, and the protection bit is dynamically offset and rounded according to the segmentation of the mantissa and the current working mode, corresponding to the current working mode and the segmentation of the mantissa.

8. The artificial intelligence processor according to claim 6, characterized in that, The dynamic offset rounding of the protection bit based on the segmentation of the mantissa and the current operating mode, corresponding to the current operating mode and the segmentation of the mantissa, further includes: in, The function definition for dynamic offset rounding, For protection purposes, For the last digit, For the mantissa width, For tensor operators, As for the current work mode, The adaptive offset adjustment function is based on the mantissa, tensor operator type, and current working mode. When the current working mode is training mode, the adaptive offset adjustment function is an alternating positive and negative offset function based on the mantissa. When the current working mode is inference mode, the adaptive offset adjustment function is a static offset function based on the mantissa.

9. The artificial intelligence processor according to claim 8, characterized in that, The adaptive offset adjustment function is: Wherein, the alternation offset coefficient k= , The scaling factor corresponds to the segmented interval, and n is the bit width adjustment variable corresponding to the tensor operator type.

10. The artificial intelligence processor according to any one of claims 6 to 9, characterized in that, The mode switching controller switches the current operating mode via a status register.

Images