Device, computer program and data carrier for quantization-based processing of activation regions in transformer layers of a neural network

Abstract

Device for executing a quantized neural network, comprising at least one computing unit configured to to divide activation data (12) of a transformer layer of the neural network into a plurality of local activation areas (16) and to process the local activation areas (16) with different quantization bit widths, wherein the respective quantization bit width of a local activation area (16) is determined as a function of an importance measure determined for the local activation area.

Description

State of the art

[0001] The invention relates to the quantized implementation of neural networks. In particular, it relates to a device, a computer program, and a machine-readable data carrier for processing activation data in transformer layers, in which different activation areas with different quantization bit widths are processed.

[0002] Modern neural networks, especially Vision Transformer (ViT) and variants such as Swin Transformer, achieve high accuracy, but are difficult to deploy for edge devices, embedded systems and AI accelerators due to the high dimensionality of activations and large computational load.

[0003] A common approach to reducing computational effort is quantization, i.e., mapping continuous activations to representations with reduced bit width. With conventional quantization methods, this reduction often has a negative impact on model accuracy.

[0004] Common methods for reducing computational effort often employ a technique called sparsity. This involves discarding or setting to zero activation values, tokens, windows, or other structural units of a neural network if they are deemed unimportant. Such sparsity or pruning methods result in certain activation areas not being processed further and thus not contributing to the model output. This can lead to information loss, broken dependency structures, and a deterioration of model accuracy. Therefore, there is a need for more efficient processing of quantized activations that appropriately handles areas of varying importance without completely discarding information.

[0005] The object of the invention is to provide an improved implementation of quantized neural networks, which makes the processing of activation data in transformer layers more efficient without affecting model accuracy. Disclosure of the invention

[0006] The task is solved by a device whose computing unit is designed to divide activation data of a transformer layer of a quantized neural network into several local activation areas and to process these local activation areas with different quantization bit widths, whereby the respective quantization bit width of a local activation area is determined depending on an importance measure specific to the respective local activation area.

[0007] The device comprises at least one computing unit configured to execute a quantized neural network. The computing unit can, for example, correspond to one of the following systems: an embedded processor, in particular, specialized inference hardware such as an NPU, DSP, or AI accelerator, a GPU, or programmable logic such as an FPGA. The device can be part of a larger system, such as a mobile device, a robot, a camera unit, a vehicle, or an edge server. The neural network executed on it comprises at least one transformer layer, for example, a self-attention layer or a subsequent MLP layer of a vision transformer architecture, ViT architecture, a Swin transformer, or any other model in which activation data exhibits spatially or logically structured regions.However, the invention is not limited to a specific Transformer variant.

[0008] According to the present disclosure, the computing unit is configured to divide activation data of a transformer layer into a plurality of local activation regions. This division enables subsequent differentiated processing of the individual regions, particularly with regard to their quantization. This allows, for example, a reduction in computational effort and / or the targeted application of precision.

[0009] Depending on the architecture or implementation, the structure can divide activation data into the following types of regions: windows, for example, rectangular or square windows of a Swin transformer; patches, as in classic ViT models; spatial regions; token groups; logical subgroups within an activation tensor field or a self-awareness matrix; or any subregions defined by the architecture. The choice of region type is not necessarily dependent on the architecture. The invention is applicable regardless of the specific transformer design.

[0010] The structure can be static, for example, with a fixed size or shape of the local activation areas; model-dependent or layer-dependent, where different layers of a neural network can have different shapes or sizes of local activation areas; or input-data-dependent, for example, through adaptive or data-driven selection of the areas, such as based on image features or statistical properties of the activations. A layer or group of layers is often referred to as a stage. Thus, an earlier stage of a neural network may have larger windows or patches than a later stage.

[0011] The importance measure is a metric derived from the activation values ​​of a given local activation area, reflecting the relative importance of that area for further processing within the neural network. The calculation of the importance measure can be based on various mathematical or statistical operations performed on the activation values, such as norm-based measures like L1 or L2 norms, statistical measures like variance, energy, or mean, evaluation functions learned from training data, or a combination of several criteria. Examples of such an importance measure include the L2 norm of a window, the variance of tokens in an area, the sum of absolute activations, activation maxima, or importance scores learned within a network.

[0012] The respective quantization bit width is defined for a local activation area depending on the importance measure determined for that area. This makes it possible to assign higher numerical accuracy to areas of higher relevance, while areas of lower relevance can be quantized with lower accuracy. Examples of possible bit width levels are 8 bits, 4 bits, 2 bits, also in combination with additional bit widths such as 6 bits, 3 bits, or other bit width variants that are hardware- or model-dependent. Flexible or dynamically determined levels can also be used.

[0013] After the respective quantization bit width has been determined, the local activation areas are further processed according to the architecture of the transformer layer. This can include, for example, quantizing the activations in the respective area, providing quantized activations for subsequent operations, and executing the steps of the transformer layer on the differently quantized areas.

[0014] In one embodiment, at least a first portion of the local activation areas is processed with a first, higher bit width, and a second portion of the local activation areas with a second, lower bit width. This design allows for the differentiated use of different quantization precisions within the same transformer layer.

[0015] The determination of the first and second proportions can be based on various criteria, such as activation characteristics, statistical properties, or an importance measure specific to the respective areas. The proportion of areas processed with higher or lower precision can vary depending on the model, the layer, or the input.

[0016] It is therefore proposed that local activation regions with a low importance score are not omitted or skipped, as is the case with activation- or token-based sparsity or pruning methods. Instead, these activation regions are further processed at a lower quantization precision. This preserves the information contained in these regions, at least with reduced accuracy. This design differs fundamentally from known methods in which activation regions are completely discarded, not considered further, or replaced by previously calculated values, which can often result in an information gap. The processing according to the invention at a lower bit width avoids this information gap and makes it possible to obtain a coherent and robust activation representation even with significantly reduced precision.Especially with transformer models featuring complex visual structures, the different precision processing helps to still consider local details of minor importance in a compressed form, resulting in a more stable overall output and improved model robustness, even when the computational effort is significantly reduced.

[0017] In further embodiments, the determination of the importance measure can be based additionally or alternatively on fixed or adaptive thresholds. These thresholds can be defined absolutely or relatively and can be, for example, percentage proportions of the activation distributions, quantified values, empirically determined limits, or thresholds learned from training data. The importance measure can also be determined by various mathematical formulas or evaluation functions. Examples include weighted norms, nonlinear activation functions, statistical parameters, energy or entropy measures, activation densities, heuristic-based parameters, or other transformation-specific metrics. The calculation of the importance measure can also be learning-based, for example, using small auxiliary networks, linear models, or evaluation functions optimized during training.

[0018] In one embodiment, the processing unit is configured to process the local activation areas in at least two processing branches with different quantization precision and to merge the outputs of the processing branches back into a common activation representation based on the original positions of the local activation areas. Each local activation area is assigned to one of the available processing branches according to its previously determined importance measure. A processing branch represents a processing path that executes the operations of the respective transformer layer with a specific quantization precision. The processing branches differ, in particular, in the quantization bit widths they use. One processing branch uses a higher bit width, while the other processing branch uses a lower bit width.

[0019] Local activation regions whose importance score indicates higher relevance to the model output can be assigned to a processing branch that uses a higher quantization bit width. Regions with a lower importance score can be assigned to a processing branch that uses a lower quantization bit width. The assignment of a region to a processing branch thus determines the quantization bit width used for processing that region.

[0020] Within each computation branch, all layer-specific operations of the respective transformer layer can be performed, such as linear transformations, matrix multiplications, activation functions, or other transformer-typical computational steps. The computation branches do not differ in the sequence of operations, but only in the quantization precision used.

[0021] After processing in the different processing branches is complete, the outputs are reassembled into a common activation representation based on the original positions of the local activation areas. For this purpose, the device can arrange the processed activation areas in the activation tensor according to their original spatial or logical arrangement, thus reconstructing the complete structure of the activation data.

[0022] In another embodiment, separate normalization layers can be provided for the different processing branches. This can be advantageous if the different quantization precisions of the processing branches lead to differing statistical distributions of the activation values. Separate normalization allows each processing branch to adjust the activation distribution appropriately for its precision, which can increase the numerical stability of subsequent processing.

[0023] In another embodiment, the computation branches can use shared network weights and each have its own quantization parameters. The computation branches access identical model weights, such as the same weight matrices of the projection, self-attention, or MLP layers. This eliminates the need to store or update the weights multiple times, and both computation branches perform the same layer-specific operations, differing only in the quantization precision used. Each computation branch has its own quantization parameters, such as step sizes, zero points, or scaling factors. These parameters determine the respective quantized representation of the activations in the corresponding computation branch and can differ from one another due to the different bit widths.For example, one processing branch handles activations with a higher bit width, while another processes activations with a lower bit width. Both processing branches use the same network weights, but with different quantization parameters.

[0024] In further embodiments, one of the computational branches, in particular the branch with higher quantization precision, can be considered the main branch, while the branch with lower quantization precision can be considered the so-called shadow branch. The shadow branch represents the same computational structure as the main branch, but uses its own quantization parameters, such as its own step sizes or zero points, while both computational branches use common network weights.

[0025] In one embodiment, the processing unit is designed such that the allocation of the respective quantization bit widths for the local activation areas and / or for a specific transformer layer in which processing takes place is dynamically determined. The bit width of the local activation areas is not fixed, but rather determined based on the available activation data or other layer-specific information. This dynamic determination can be performed for individual local activation areas as well as for a specific transformer layer as a whole. This allows the bit width to adapt to different inputs, activation patterns, or internal states of the neural network. The dynamic determination can be based on a previously determined importance measure, on statistical properties of the activations, or on layer-specific parameters.The bit width selection is made at runtime and / or depending on the current data and is not statically defined.

[0026] In one embodiment, the dynamic determination of the respective quantization bit widths for the local activation regions in different transformer layers of the neural network is based on different criteria and / or parameters. The dynamic determination of the quantization bit widths can therefore be layer-specific. Different criteria and / or parameters can be used for different transformer layers to dynamically determine the bit width for the local activation regions. The dynamic determination can differ, for example, in how activation values ​​are evaluated, which weights or thresholds are applied, or which quantization-specific parameters are relevant for the respective layer.This allows the device to adapt the choice of quantization bit width to layer-specific characteristics such as activation distributions, model structures, or resolution levels.

[0027] In one embodiment, the device is configured to perform quantization-aware training (QAT) and / or post-training quantization (PTQ), using the importance measure determined for the respective local activation regions to dynamically derive the corresponding quantization bit widths. In quantization-aware training, the subsequent quantized execution of the neural network is already taken into account during the training phase, allowing the model parameters to be adjusted to compensate for quantization errors. In post-training quantization, an already trained model is converted into a quantized form using calibration data.In both cases, the importance measure determined for the respective local activation regions can be used to dynamically derive the corresponding quantization bit widths during training or calibration. This allows the precision distribution to be adapted to the importance of the individual activation regions during the optimization process itself. This can contribute to improving the accuracy and robustness of the quantized model. During a training procedure, gradient signals from both computational branches can be accumulated across the shared network weights. These gradient signals can originate, for example, from stochastic gradient calculations, methods of moments, or other optimization techniques. Sharing the network weights enables parameter changes to be made based on the gradients calculated in both computational branches.Furthermore, a modified loss function can be used during a training process to influence the dynamic bit width allocation or the determination of the importance measure. The loss function can, for example, include additional terms or regularizations that take into account the stability of the quantization, the bit width distribution, or the sensitivity to activation values.

[0028] Another embodiment relates to a computer program with instructions which, when executed on a computing unit, cause the computing unit to operate a device according to the described embodiments. The computer program can be configured to control all steps necessary for the operation of the device, including the structuring of the activation data, the determination of an importance measure, and the quantized processing of the local activation areas. The computer program can be executed on any programmable computing unit, such as an embedded processor, a GPU, an NPU, or an AI accelerator.

[0029] The processing of activation areas can be hardware-optimized, for example, using SIMD or vector instructions, parallel processing units, tile-based processing, or pipeline-based data paths. The allocation of activation areas to the compute branches can also be supported by timing control, thread-based scheduling, or workload-dependent distribution.

[0030] In a further embodiment, the computer program comprises instructions that cause the processing unit to divide the activation data of a transformer layer into a plurality of local activation regions and to determine an importance measure for each local activation region based on the respective activation values. The instructions control both the division of the activation data into local activation regions and the calculation of the respective importance measure based on the activation values ​​present in the local activation region.

[0031] In another embodiment, the computer program includes instructions that cause the processing unit to define a specific quantization bit width for each local activation area, depending on the defined importance measure, and to process the local activation areas in at least two processing branches with different quantization precision. The instructions can control the bit width allocation for each local activation area and its assignment to the processing branches, with one processing branch being provided with higher quantization precision and another with lower quantization precision.

[0032] In another embodiment, the computer program includes instructions that cause the processing unit to reassemble the outputs of the processing branches into a common activation representation based on the original positions of the local activation areas. The instructions can be configured to insert the results from the different processing branches into the original spatial or logical arrangement of the local activation areas within the activation tensor, thereby generating a complete activation representation of the transformer layer.

[0033] In one embodiment, the computer program includes instructions that cause the processing unit to dynamically determine the allocation of the respective quantization bit width for the local activation regions and / or for a specific transformer layer in which processing takes place. The instructions can be configured to adapt the bit width selection during processing to the current activation values ​​or layer-specific properties, thereby enabling dynamic determination of the quantization bit width. This dynamic determination can be performed for individual local activation regions as well as for an entire transformer layer, allowing for flexible adaptation to different activation patterns or internal states of the neural network.

[0034] In another embodiment, the computer program includes instructions that instruct the processing unit to use different criteria and / or parameters for dynamically determining the respective quantization bit widths for the local activation regions of different transformer layers of the neural network. The instructions can take layer-specific differences into account and adapt the dynamic bit width determination to the respective properties of the transformer layers, so that the selection of the quantization bit width does not have to be identical in all layers. This allows the processing unit to adapt the bit width selection to different activation structures or internal characteristics of the transformer layers.

[0035] In another embodiment, the computer program includes instructions that instruct the processing unit to execute the program within the framework of quantization-aware training (QAT) and / or post-training quantization (PTQ), using the specified importance measure for dynamic bit width allocation. The instructions may stipulate that during quantization-aware training, the quantized execution of the neural network is taken into account, so that the model parameters are adjusted to the resulting quantization errors. Furthermore, the instructions may stipulate that in post-training quantization, an already trained model is converted into a quantized form using calibration data, and the specified importance measure is used to dynamically derive the respective quantization bit widths.The instructions thus make it possible to adapt the distribution of the quantization bit widths to the importance of the individual local activation areas even during a training or calibration process.

[0036] According to a further embodiment, the invention comprises a machine-readable data carrier on which a computer program according to the described embodiments is stored. The data carrier can be volatile or non-volatile memory, for example, flash memory, a solid-state drive, a memory card, or a RAM-based memory module. When the computer program stored on the data carrier is loaded and executed by a processing unit, the instructions contained therein cause the processing unit to execute the steps encompassed by the instructions of the computer program.

[0037] Further advantages will become apparent from the description and the accompanying drawing. Exemplary embodiments of the invention are shown in the drawing and are explained in more detail in the following description. Here, identical reference numerals in different figures denote identical or at least functionally comparable elements. When describing individual figures, reference is also made, where appropriate, to elements from other figures. It shows in schematic form: Fig. Figure 1 shows a diagram of a block of a transformer model set up for quantized processing of local activation areas.

[0038] Fig. Figure 1 shows a block diagram of block 10 of a transformer model, which is traversed during the execution of a quantized neural network. The depicted block 10 processes activation data 12, which is associated with a transformer layer of the neural network. The activation data 12 is taken from a preceding block 14 and, after processing, passed on to a subsequent block 15.

[0039] In a prior art transformer model, a block also processes activation data from a preceding block and passes a suitable representation of the processed data to a subsequent block. However, prior art transformer models do not perform quantized processing based on an importance measure defined for a specific local activation area. Quantized processing is explained below in relation to the present invention.

[0040] The depicted block 10 could, for example, correspond to a Swin block as used in Swin transformer architectures. Likewise, the block could be associated with another transformer model that has a comparable internal structure of attention and feed-forward operations.

[0041] The in Fig. Block 10 shown divides the incoming activation data 12 of a transformer layer into a plurality of local activation areas 16.

[0042] The local activation areas 16 are windows, patches, regions, or token groups of the transformer layer. In this example, the local activation areas 16 are windows, as used in Swin transformer architectures.

[0043] For each local activation area 16, an importance measure is determined.

[0044] The sequence of local activation areas 16 of the activation data 12 shown is merely exemplary and does not depend on the importance measure determined for each local activation area 16.

[0045] The importance measure can be determined, for example, based on the respective activation values ​​of a given local activation area 16. Examples of such an importance measure include norm-based measures, such as the L1 norm or the L2 norm of the activation values, statistical parameters such as variance or energy, or the sum of the absolute activation values.

[0046] Based on the importance measure, 16 different quantization bit widths are assigned to the local activation areas, so that the activation areas can be further processed with different quantization precision.

[0047] In the example, the activation areas 16 are divided into a first group 18 of activation areas 16, in particular windows, with a high importance level and into a second group 20 of activation areas 16, in particular windows, with a lower importance level.

[0048] In this example, the activation data in block 10 is processed such that the first group 18 of the local activation areas 16 is processed with a first, higher bit width and the second group 20 of the local activation areas 16 is processed with a second, lower bit width.

[0049] According to the example, the one in Fig. Block 10, as shown in Figure 1, is set up to assign the local activation areas 16 to different computation branches based on a specific importance measure. In this example, block 10 comprises at least two computation branches 30 and 32, which are characterized by different quantization precision.

[0050] The first processing branch 30 is configured to process the first group of 18 local activation areas 16 with a higher quantization bit width. The second processing branch 32 is configured to process the second group of 20 local activation areas 16 with a lower quantization bit width. Each local activation area 16 is assigned to either processing branch 30 or processing branch 32 according to its importance.

[0051] Within processing branches 30 and 32, the computational steps typical for the respective transformer layer are executed. The sequence of computational steps is the same in both processing branches, which differ only in the quantization precision used. By dividing the activation areas 16 between processing branches 30 and 32, processing within the same block 10 can be performed at different bit widths. This two-part processing of the processing branches can be applied to both the attention operation and the feed-forward operation of a transformer layer.

[0052] After processing in the computation branches 30 and 32 is complete, the respective outputs 34 are recombined based on the original positions of the local activation areas 16 in the activation data 12. For this purpose, the activation areas 16 are inserted in their original spatial or logical arrangement within the activation data 12, so that a common activation representation 36 is generated, which represents the fully processed activation data of block 10.

[0053] In the example, the computation branches 30 and 32 are configured to use shared network weights 17 and 19. Both computation branches 30 and 32 access the same weight matrices of the transformer layer, for example, the weight matrices of the attention operation or the subsequent feed-forward operation. This eliminates the need to store or update the network weights 17 and 19 multiple times, while the processing in both computation branches 30 and 32 involves identical computational operations.

[0054] The computation branches 30 and 32 merely form different quantized representations of the same network operations, without generating separate sets of weights for this purpose.

[0055] Although network weights 17 and 19 are shared, each computing branch 30 and 32 has its own quantization parameters. These parameters include, in particular, step sizes, zero points, and scaling factors, which determine how the activation values ​​are quantized and processed in the respective computing branch 30 and 32. The quantization parameters may differ due to the different quantization bit widths of computing branches 30 and 32. Furthermore, the quantization parameters can be managed separately for the activations and, where provided, for the shared weights 17 and 19.

[0056] In this example, each computation branch 30, 32 comprises its own normalization layers 22, 23, 24, 25. The normalization layers 22, 23, 24, 25 in the two computation branches 30, 32 are structurally identical but are executed separately and each has its own parameters, since the distributions of activation values ​​in computation branches 30, 32 differ due to the different quantization precision. This allows each computation branch 30, 32 to normalize the activation values ​​processed within it according to the quantization precision used there. The separate normalization layers 22, 23, 24, 25 can be placed in block 10 before the attention operation 26, 27 of block 10 and before the subsequent feed-forward operation 28, 29 of block 10.

[0057] According to the example, the first processing branch 30 comprises a first normalization layer 22, also known as LN high, an attention operation 26, also known as MHA high, another normalization layer 24, also known as LN high, and a feed-forward operation 28, also known as FFN high. The second processing branch 32 accordingly comprises a first normalization layer 23, also known as LN low, an attention operation 27, also known as MHA low, another normalization layer 25, also known as LN low, and a feed-forward operation 29, also known as FFN low.

[0058] The attention operations 26, 27 of the processing branches 30, 32, namely MHA high and MHA low, and the feed-forward operations 28, 29 of the processing branches 30, 32, namely FFN high and FFN low, use identical network weights 17, 19, but differ in the quantization bit width used for processing and in separate quantization parameters such as step sizes and zero points. The separate normalization layers 22, 23, 24, 25, namely LN high and LN low, make it possible to adapt the statistical distribution of the activation values ​​to the quantization precision used in the processing branch.

[0059] The outputs 34 generated in both computation branches are combined after processing into a common activation representation 36 based on the original positions of the local activation areas 16.

[0060] A prior art method for processing activation data using a prior art transformer model can be based on the following architecture: A preceding block provides activation data. This data is processed in the transformer block, where the processing includes a first normalization layer, an attention operation, a second normalization layer, and a feed-forward operation. The processed activation data is then passed to a subsequent block in a suitable representation. However, the prior art transformer model does not assign data to different processing branches with varying levels of quantization precision based on an importance measure.

Claims

[1] Device for executing a quantized neural network, comprising at least one computing unit configured, to divide activation data (12) of a transformer layer of the neural network into a plurality of local activation areas (16) and to process the local activation areas (16) with different quantization bit widths, wherein the respective quantization bit width of a local activation area (16) is determined as a function of an importance measure determined for the local activation area. [2] Device according to claim 1, wherein the local activation areas (16) comprise windows, patches, regions or token groups of the transformer layer. [3] Device according to one of the preceding claims, wherein the computing unit is designed such that the importance measure is calculated from activation values ​​of the respective activation area (16). [4] Device according to one of the preceding claims, wherein the computing unit is designed such that at least a first part (18) of the local activation areas (16) is processed with a first, higher bit width and a second part (20) of the local activation areas (16) is processed with a second, lower bit width. [5] Device according to one of the preceding claims, wherein the computing unit is configured to process the local activation areas (16) in at least two computing branches (30, 32) of different quantization precision, and to combine the outputs (34) of the computing branches back into a common activation representation (36) based on the original positions of the local activation areas (16). [6] Device according to claim 5, wherein the computing unit is configured such that the computing branches (30, 32) use shared network weights (17, 19) and each have their own quantization parameters. [7] Device according to one of the preceding claims, wherein the computing unit is configured such that the allocation of the respective quantization bit widths for the local activation areas (16) and / or for a respective transformer layer in which the processing takes place is dynamically determined. [8] Device according to claim 7, wherein the dynamic determination of the respective quantization bit widths for the local activation areas (16) in different transformer layers of the neural network is based on different criteria and / or parameters. [9] Device according to one of the preceding claims, wherein the device is set up to perform quantization-aware training and / or post-training quantization and uses the importance measure determined for the respective local activation areas (16) for the dynamic derivation of the associated quantization bit widths. [10] Computer program comprising instructions which, when executed on a computing unit, cause the computing unit to operate a device according to any one of claims 1 to 9. [11] Computer program according to claim 10, comprising instructions that cause the computing unit to, to divide activation data of a transformer layer into a plurality of local activation regions (16), and to determine an importance measure for each local activation area (16) based on the respective activation values. [12] Computer program according to one of claims 10 or 11, comprising instructions that cause the computing unit to determine a respective quantization bit width for each local activation area (16) depending on the determined importance measure, and to process the local activation areas (16) in at least two computing branches (30, 32) of different quantization precision. [13] Computer program according to one of claims 10 to 12, comprising instructions that cause the computing unit to combine the outputs (34) of the computing branches (30, 32) into a common activation representation (34) based on the original positions of the local activation areas (16). [14] Computer program according to one of claims 10 to 13, comprising instructions that cause the computing unit to dynamically determine the allocation of the respective quantization bit width for the local activation areas (16) and / or for a respective transformer layer in which the processing takes place. [15] Computer program according to one of claims 10 to 14, comprising instructions that cause the computing unit to use different criteria and / or parameters for different transformer layers of the neural network to dynamically determine the respective quantization bit widths for the local activation areas (16). [16] Computer program according to any one of claims 10 to 15, comprising instructions that cause the computing unit to execute the program as part of quantization-aware training and / or post-training quantization, using the specified importance measure for dynamic bit width allocation. [17] Machine-readable data carrier on which a computer program according to any one of claims 10 to 16 is stored.

Images