Relevance score assignment for machine learning predictors with attention modules and / or softmax function

Abstract

Apparatus for assigning relevance scores to portions to be evaluated of a machine learning (ML) predictor including one or more attention modules, the apparatus configured to determining the relevance scores by back-propagating an initial relevance score at an output of the ML predictor or at internal portion of the ML predictor, on the basis of activations of the ML predictor, which manifest itself in an performed by the ML predictor in portions of the ML predictor, which lie upstream relative to the output or the internal portion along an activation direction of the ML predictor, and include the one or more attention modules. Additional apparatuses utilizing the proposed apparatus as well as a corresponding method and computer program are also disclosed.

Description

[0001]Relevance Score Assignment for Machine Learning Predictors with Attention Modules and / or Softmax Function Technical Background The present application is concerned with relevance score assignment for a machine learning predictor such as artificial neural networks with attention modules and / or a softmax function. Such relevance score assignment may be used, for example, for region of interest (ROI) identification. The attention mechanism (Vaswani et al., 2017) became an essential component of large transformers due to its unique ability to handle multimodality and to scale to billions of training samples. While these models demonstrate impressive performance in text and image generation, they are prone to biased predictions and hallucinations (Huang et al., 2023), which hamper their widespread adoption. To overcome these limitations, it is crucial to understand the latent reasoning process of transformer models. Researchers started using the attention mechanism of transformers as a means to understand how input tokens interact with each other. Attention maps contain rich information about the data distribution (Clark et al., 2019; Caron et al., 2021), even allowing for image data segmentation. However, attention, by itself, is inadequate for comprehending the full spectrum of model behavior (Wiegreffe and Pinter, 2019). Similar to latent activations, attention is not class-specific and solely provides an explanation for the softmax output (in attention layers) while disregarding other model components. Recent works (Geva et al., 2021; Dai et al., 2022), e.g., have discovered that factual knowledge in Large Language Models (LLMs) is stored in Feed-Forward Network (FFN) neurons, separate from attention layers. Further, attention-based attribution methods such as rollout (Abnar and Zuidema, 2020; Chefer et al., 2021a) result in checkerboard artifacts, e.g., for a Vision Transformers (ViTs). Researchers thus have turned to model-agnostic approaches that aim to provide a holistic explanation of the model’s behavior (Miglani et al., 2023), including, e.g., perturbation and gradient-based methods. Methods based on feature perturbation require excessive amounts of compute time (and energy), making their application uneconomically, especially for large architectures. Gradient-based methods, on the other hand, are highly efficient but suffer from noisy gradients and low faithfulness, as evaluated in Section 4.1. Consequently, for understanding latent neurons, perturbation methods are often the preferred choice (Meng et al., 2022; Bills et al., 2023). Another option is to take advantage of the versatility of rule-based backpropagation methods, such as Layer-wise Relevance Propagation (LRP). These methods allow for the customization FH240201PEP-2025036018.DOCX of propagation rules to accommodate novel operations, allowing for more faithful explanations and requiring only a singular backward pass. Attempts to apply LRP to non-linear attention, however, have faced challenges such as numerical instability, lack of faithfulness, or completely circumventing the non-linear softmax problem. Therefore, there is still a need for a robust, easy to implement and widely applicable concept for realizing the task of relevance score assignment for machine learning (ML) predictors comprising attention modules. Accordingly, it is an object of the present invention to provide a concept for assigning a relevance score to portions of a ML predictor comprising an attention module and / or a softmax function. The proposed methods should be applicable to a broad set of ML predictors, provide faithful relevance scores and / or present low computational efforts. This object is achieved by the subject matter of the pending independent claim. Summary of the Invention An initial aspect of the invention presents an apparatus for assigning relevance scores to portions to be evaluated of a machine learning (ML) predictor, also called ML predictor, including one or more attention modules and / or a softmax function. The apparatus is configured to determine the relevance scores by back-propagating an initial relevance score at an output of the ML predictor or at internal portion of the ML predictor (e.g., the activation at the output or the internal node, which is obtained by an inference performed by the ML predictor), on the basis of activations of the ML predictor. The activations manifest itself in an inference (e.g., the inference which yielded the initial relevance score) performed by the ML predictor in portions of the ML predictor, which lie upstream relative to the output or the internal portion along an activation direction of the ML predictor, and include the one or more attention modules and / or the softmax function.] In particular, the inventors of the present application found out a way to back-propagate the initial relevance score through ML predictors that include one or more attention modules and the resulting insight forms the basis for the embodiments presented in the present application. By incorporating attention modules in the relevance score back-propagation, the apparatus is able to achieve numerically correct and faithful relevance scores with negligible additional computational costs. However, the embodiments of the present application are capable of, despite the presence of such one or more attention modules, or differently speaking by FH240201PEP-2025036018.DOCX traversing the one or more attention modules, achieve the back-propagation. That is, the inventors found a way that the back-propagation may, in a meaningful manner, be continued across the one or more attention modules or parts thereof. In existing concepts for relevance back-propagation, the attention modules of machine learning (ML) predictors are either excluded or approximated, resulting in less faithful and numerically unstable relevance scores. Furthermore, the individual components of an attention module, such as nodes and their interconnections, are often assigned with low faithfulness and high numerical instability when relevance scores are provided at all. According to embodiments, numerically stable and faithful relevance scores are assigned to every part of the ML predictor, including all individual components of the attention modules; this is, according to these embodiments, achieved by applying certain rules the inventors have found out for relevance back-propagation across various components and processing steps within attention modules, such as sigmoid functions and dot products. The obtained relevance score can, for example, be used to prune a neural network / ML predictor. When pruning neural networks, parts of the neural network are discarded. For example, by setting some node connections to zero, some are effectively cut. An apparatus according to an embodiment of this invention can obtain good relevance scores for every single part of a neural network including parts of the one or more attention modules. These relevance scores may then be used to decide as to which parts of the neural network should be discarded. This process requires good relevance scores, but in return can lead to significantly reduced computational costs for a forward pass of the pruned neural network, when compared to the pre-pruned neural network. According to an embodiment, the ML predictor comprises nodes and node interconnections and the apparatus is configured to determine the relevance scores for the nodes and / or the node interconnections of the ML predictor by back propagating the initial relevance score by distributing a relevance score at a predetermined node of the ML predictor onto predecessor nodes of the predetermined node according to, for each of the predecessor nodes, a fraction associated with the respective predecessor node, and determining, for each predecessor node, the fraction associated with the respective predecessor node depending on a further fraction at which the activation of the respective predecessor node contributes to the activation of the predetermined node in the inference. FH240201PEP-2025036018.DOCX According to an embodiment, the relevance score for a predetermined portion of the ML predictor, which is composed of more than one node and / or node inter connection of the ML predictor, is determined by aggregating the relevance scores of the more than one node and / or node interconnection the predetermined portion is composed of. E.g., this might be done by inspecting the summation of inbound relevance score distributions from successor nodes in layer l+1 to a certain node in layer l, with counting the layer indices l in a manner ascending in forward-propagation sense, i.e., along the inference direction. An embodiment of the invention describes an apparatus that is, configured to sum-up, according to a summation function, the relevance scores distributed from successor nodes of a further predetermined node so as to obtain the relevance score for the further predetermined node. In another embodiment, the apparatus is configured to sum-up if the further predetermined node is contained in a linear layer of the ML predictor. According to an embodiment, back-propagating the relevance scores is performed by back- propagating the relevance scores at successor nodes of a softmax function (e.g., of a predetermined attention module) onto a predetermined predecessor node of the softmax function, by forming a product between 1) a difference between a) a summation over the relevance scores at the successor nodes of the softmax function, multiplied with the softmax function`s output for a corresponding successor node, associated with the predetermined predecessor node, as obtained in the inference, and b) the relevance score of the corresponding successor node, and 2) the softmax function`s input for the predetermined predecessor node as obtained in the inference. Exemplarily the equation (13) can be used. As mentioned above, using the concepts described in embodiments of the present application, softmax functions may be crossed in back-propagation. The non-linearity of the softmax function represents a challenge when applying relevance back-propagation to ML predictors comprising an attention module. Since the attention modules may comprise a softmax function, the attention module is usually left out, when using relevance back-propagation on ML predictors comprising attention modules. However, using the described rule for relevance back-propagation can allow to extend the relevance back- propagation to incorporate, or traverse, the attention module. This can lead to the concept being applicable to a wider range of ML predictors as well as to overall better faithfulness of the acquired relevance scores, since all parts of the ML predictor have been taken into account during the relevance back-propagation, while incurring negligible computational resources. FH240201PEP-2025036018.DOCX According to an embodiment, back-propagating the relevance scores involves back- propagation of the relevance scores at successor nodes of a dot product function of a predetermined attention module onto a predetermined predecessor node of the dot product, by using equation (15). According to an embodiment, during back-propagating the relevance scores, the back- propagation is only continued for relevance scores with an absolute value greater than a certain threshold and / or with an absolute value greater than zero. Due to the typically present sparsity of relevance scores, such an embodiment can assign the relevance scores with less computations. This only slightly decreases the accuracy of the assigned relevance scores, however, the improved computational cost associated with it can represent a better compromise between accuracy and computational cost. According to further aspects of the invention, the above concepts are used to result in an apparatus for assigning relevance scores to context inputs of a ML predictor including one or more attention modules. Examples for such context inputs encompass user provided data, database data, documents, legal or other texts, facts, or portions, paragraphs or sections of documents. The relevance scores are indicative of a relevance of the context inputs for an output of the ML predictor output by the ML predictor when fed with a query along with the context inputs. In other words, the above described concepts are used here for the relevance assignment to context inputs. Further examples for context inputs include image, video and audio data, wherein relevance score can, for example, be assigned to specific pixels of images, regions of images, pixels of videos, pixels of videos over a timeframe, regions of videos over a period, periods of videos of periods of audio data. In other words, according to this further aspect, the relevance scores, obtained using the above concepts of the initial aspect, can be used to measure the significance individual context inputs have on the output. This can, for example, be used to highlight context inputs. For example, in autonomous driving, the apparatus can be configured to highlight dangerous parts of the road using a heads-up display. For such an implementation, the ML predictor would receive a video feed of the road as input and categorize the road into categories such as including dangerous or not dangerous. According to an even further aspect of the invention, an apparatus for pruning and / or quantizing an ML predictor is obtained by using the relevance score assignment according to the initial aspect. Such apparatus may be configured to, using the relevance scores obtained FH240201PEP-2025036018.DOCX as described according to the initial aspect, control a pruning and / or quantizing of the ML predictor. Relevance scores, obtained using an apparatus according to the initial aspect of the invention, can be used to prune or quantize the ML predictor. This can for example be used to reduce computation power during the inference of the ml predictor. The apparatus can for example be configured to deactivate certain parts of the ML predictor and not run them at all and in return save time and computation power needed for inference. Brief description of the drawings The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which: Fig.1a-e show schematic views of an apparatus, according to the initial aspect of the invention, assigning relevance scores; Fig.2 shows a broad comparison of the capabilities of an apparatus according to an embodiment of the invention with SmoothGrad, GradxAttnRoll and AtMan; Fig.3a-b show an example of manipulating the activation of individual neurons using relevance scores obtained using an apparatus according to embodiments of the invention; Fig.4 shows a table, that compares the IoU of different methods to an apparatus according to embodiments of the invention on different transformer ML predictors; Fig.5 shows a table, that compares the computational complexity and memory complexity of LRP-based and liner-time perturbation methods; Fig.6 shows a comparison of ^^AttnLRP (e.g., ^^Attn-LRP) with GradൈAttnRoll; Fig.7a-d show evaluations of faithfulness for different methods on different datasets; Fig.8 shows a table with an overview of proposed composites; Fig.9a-e show statistics for different layer type, comparing different relevance propagation rules; Fig.10a-c illustrate the top 10 sentences in the Wikipedia summary dataset that maximally activate a knowledge neuron; Fig.11 shows a schematic view of an apparatus according to the even further aspect of the invention, that prunes and / or quantizes a ML predictor; Fig.12a-b show schematic views of an apparatus according to the further aspect of the invention, that highlights portions of the context input; FH240201PEP-2025036018.DOCX Fig.13 shows a table 1380, comparing different methods and their approach to different operations of a ML predictor; Fig.14 shows an apparatus, controlling an activation filtering functionality of a ML predictor; Fig.15 shows an apparatus that influences the activation of a specific neuron of a ML predictor during inference; Fig.16 shows an apparatus that provides a certainty measure for an output of a ML predictor; and Fig.17 shows an apparatus for auditing a ML predictor. Detailed Description of the embodiments Equal or equivalent elements or elements with equal or equivalent functionality are denoted in the following description by equal or equivalent reference numerals even if occurring in different figures. It is to be noted, that in this description an indexing X1-krelates to elements X1- Xk, or in other words X1to Xk.In the following description, a plurality of details is set forth to provide a more thorough explanation of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring embodiments of the present invention. In addition, features of the different embodiments described herein after may be combined with each other, unless specifically noted otherwise. Although some Figures relate to images and pixels, embodiments of the present application are not restricted to that kind of data. Similarly, the ML predictor is merely exemplified as a model or neural network in the following description. Fig. 1a shows a schematic view of an apparatus 100 according to the initial aspect of the invention. The apparatus is configured to assign relevance scores 103 to portions 102 to be evaluated of a machine learning (ML) predictor 101. The (ML) predictor 101 includes one or more attention modules and / or a softmax function 115. The apparatus 100 can be configured to determine the relevance scores 103 by back-propagating an initial relevance score 125 at the output 110 of the ML predictor 101 or at an internal portion of the ML predictor 101 on the basis of activations 104 of the ML predictor 101. The activations 104 manifest itself in an inference performed by the ML predictor 101 in portions 123 of the ML predictor 101, which lie upstream relative to the output 110 or the internal portion along an activation direction 105 of FH240201PEP-2025036018.DOCX the ML predictor 101, and include the one or more attention module 122 and / or softmax function 115. The activation direction 105 can also be called inference direction. Thus, this is the direction along which the ML predictor passes along the activations during the inference, while the back-propagation is performed contrary to this direction, sometimes called back- propagation direction in the following. The internal portion can be downstream along the activation direction 105 relative to the portion 123 with the portion being positioned, for instance, between the portions 102 to be evaluated 102. The ML predictor 101 is configured to, in the inference, upon being fed with some input signal at an input of the ML predictor output an output signal at the output 110. The portions 102 can, for example, be a single node, a layer of nodes, a collection of nodes, a part of a convolutional layer, part of a further attention module, part of the attention module 122, a matrix or any other part of the ML predictor 101, that influences the ML predictor 101 during the inference. The attention module 122 and / or softmax function 115 can lie upstream along the activation direction 105 compared to the portion 102 to be evaluated as illustrated in Fig. 1a, or downstream. The ML predictor 101 can, for example, be an LLM or a Vision Transformer (ViT). It may be configured to receive text, pictural data, video data, and / or audio data as an input signal and output the same sort of data or a different collection of one or more out of text, pictural data, video data, and / or audio data as an input signal at its output. More examples are given at a later stage. The sort of task fulfilled by the ML predictor on the input signal might be such that the output sign forms an answer to a question represented by the input signal, or may be the generation of the output signal The apparatus of Fig.1a can be extended by any of the Figures 1b-e in terms of functionality individually or in combination. Fig. 1b-e schematically show the apparatus 100 with each of Fig. 1b-e showing different exemplary parts of the ML predictor 101 to explain certain functionalities of the apparatus. In particular, Fig.1b-e exemplarily show different details of the back-propagation of relevance scores 103 along the back-propagation direction 124. In Fig. 1b, an exemplary section out of the ML predictor 101, comprising nodes and node interconnections 119, is shown. The interconnections 119 are define those interconnections along which the nodes of the ML predictor mutually forward their activations within a forward pass of inference performed by the ML predictor 101, controlled via corresponding weights defining the ML predictor 101, one per interconnection 119. The illustrated section can, for example, relate to linear layers and / or convolutional layers. Three predecessor nodes 1071-3are shown, that are connected through interconnections 1191-3with a predetermined node 106. The predecessor nodes 1071-3precede the predetermined node 106 along direction 105. Thus, FH240201PEP-2025036018.DOCX during inference, the predecessor nodes 1071-3forward their activations, weighted by the ML predictor’s weights to the predetermined node 106. The interconnections 119 are shown as arrows in order to reflect the direction along which the predecessor nodes’ 1071-3activations are forwarded to the predetermined node 106. At node 106, the inbound activations are summed up and the predetermined node 106 forwards, guided by respective weights, its activation to respective successor nodes 121 which, thus, follow the predetermined node 106 along direction 105 and receive activations from the predetermined node via corresponding interconnections depicted, again, as arrows pointing into activation forward direction. The predecessor nodes 1071-3and predetermined node 106 each experience an activation 1041-3(A1-3) during an inference of the ML predictor 101. These activations are used to guide or control the relevance score back-propagation which starts with a relevance score at the output 110 or the internal portion mentioned above, with this relevance score being derived from, or set to be equal to, the activation manifesting itself at this output 110 or the internal portion during the inference. Note that instead of using activation manifesting itself during one inference, the activations used may be gained by averaging over activations experienced, on average, during a plurality of inferences. During back-propagation, a relevance score 103 is assigned to or associated with the predetermined node 106, namely by summing up all relevance score fractions inbound from the successor nodes 121. Fig. 1b concentrates on the back-propagation of node’s 106 relevance score to the predecessor nodes 1041-3, i.e., on the explanation as to how the relevance score of nodes is portioned or fractionized in order to backwardly forward the resulting fractions to the predecessor node 1041-3. In other words, the apparatus 100 can be configured to determine the relevance scores 1031-3for the nodes and / or the node interconnections of the portion 102 to be evaluated by back propagating the initial relevance score 125 from the output 110 or the internal portion, and this is done by distributing the relevance score 103 at the predetermined node 106,as obtained by summation of the inbound score fractions from successor nodes 121, onto the predecessor nodes 1071-3according to, for each of the predecessor nodes 1071-3, a fraction associated with the respective predecessor node 1071-3. The apparatus 100 can further be configured to determine, for each predecessor node 1071-3, the latter fraction associated with the respective predecessor node 1071-3depending on a further fraction at which the activation 1041-3of the respective predecessor node 1071-3contribute to the activation 104 of the predetermined node 106 in the inference. That is, the fraction at which node’s 106 relevance score is back- propagated to a certain one of the predecessor nodes 1041-3depends on the ratio between the activation of this certain predecessor node on the one hand and the sum of all predecessor FH240201PEP-2025036018.DOCX node’s 1041-3activations. Again, Fig.1b concentrates on one node 106, but apparatus 100 is configured to perform this back-propagation for each of the nodes traversed by the back- propagation from the output 110 or the internal portion to the portion 102 to be evaluated. The predetermined node 106 exemplarily depicted in Fig. 1b, can, for example, be part of the portion 123 of the ML predictor 101 and the predecessor nodes can be part of the portion 123, too, or the portion 102. In Fig. 1b, successor nodes 1211– 121kof the predetermined node 106 can, for example, represent a layer l+1 of the ML predictor, while the predetermined node 106 is part of a layer l and the predecessor nodes are part of a layer l-1. As described, the apparatus 100 can be configured to sum-up, according to a summation function, the relevance scores 103s1- 103skdistributed from successor nodes 1211– 121kof a certain node to obtain the relevance score of the certain node, and this done also, for example, for node 106 based on nodes’ 121 relevance score fractions and for any of the predecessor nodes 104 based on the inbound relevance score fractions including the one received from node 106. The apparatus 100 can be configured such that the (e.g., pure) summation over the inbound relevance scores 103s1-kor inbound relevance score fractions 103s1-kdistributed from corresponding successor nodes 1211-kis free of an additive offset. For example, equation (8) might be used. The apparatus 100 can also be configured such that, if the ML predictor 101 is a Vision Transformer, check whether the successor nodes (121) are part of a linear layer or a convolutional layer of the ML predictor (101) and if the successor nodes (121) are part of a linear layer or a convolutional layer of the ML predictor (101), perform the summation over the relevance scores (103) distributed from the successor nodes (121) using a ^^-rule or a ^^^-rule if the successor nodes (121) are not part of a linear layer or a convolutional layer of the ML predictor (101), perform the summation over the relevance scores (103) distributed from the successor nodes (121) in a manner free of an additive offset. In Fig. 1c, an exemplary section of the ML predictor 101 is shown, which represents a dot product 114 application in the ML predictor 101 involved in the inference. Such section can, for example, be part of an attention module of the ML predictor 101 such as the attention module 122 of the ML predictor. or any other attention module of the ML predictor 101. Alternatively, the section may outside any attention module. FH240201PEP-2025036018.DOCX The dot product 114 is shown illustratively by use of the underlying mathematics. The dot product 114 is applied on an operand 111 and a further operand 112, resulting in a matrix 113. The matrix 113, the operand 111 and the further operand 112 stand exemplary for any matrices or a scalar in any part of the ML predictor 101. The operand 111 can, for example, be an attention matrix A, the further operand can, for example, be a value matrix V, and the matrix 113 can, for example, be a context matrix. The operand 111 and further operand 112 can, for example, be part of the portion 102 that is to be evaluated, or may be part of portion 123 and the matrix 113 can, for example, be part of the portion 123. Activations 104A: A11, A12, A21and A22of the operand 111 and activations 104V: V11, V12, V21, V22, V31and V32of the further operand 112, as they are experienced during inference, are shown. During inference, these activation result in the activations 104M:Ml11, Ml12, Ml13, Ml21, Ml22and Ml23of the matrix 113. During back-propagation, each node of the matrix 113 can have a relevance score 103lassociated with it (e.g., ^^^^^ , ^^^^ଶ , ^^^^ଷ , ^^ଶ^^ , ^^ଶ^ଶ and ^^ଶ^ଷ ). Fig. 1c illustrates, as to how these relevance scores might be back propagated across the dot product section. The elements of the matrix 113 represent successor nodes 121 of the dot product 114 and elements of the operand 112 and the further operand 111 represent predecessor nodes 107 of the dot product 114. Successor nodes 121 and predecessor nodes 107 are nodes of the ML predictor. According to Fig. 1c, the apparatus 100 can be configured to, in back-propagating the relevance scores 103lacross the dot product section, back-propagate the relevance scores 103lat successor nodes 121 of the dot product 114 function (e.g., of a predetermined attention module) onto a predetermined predecessor node (e.g., any one of the nodes of the operand 111) of an operand 111 of the dot product 114, by using the equationோ^ ೕ^^^^^^^^^^ଶ ைೕ^^ఌ, with ^^^^ൌ∑^^^^^^^^^and ^^ being a number near zero with the same sign as ^^^^(if ^^^^= 0, then the sign is positive) or ^^ being zero and ^^^^ is the index of the predetermined predecessor node . In other words, the apparatus 100 can also be configured to back-propagate the relevance scores 103lat successor nodes 121 of a dot product 114 function of an operand 111 and a further operand 112, onto a predetermined predecessor node 107 of the dot product 114, by forming a sum over summands, that are a product of 1) an activation of the predetermined predecessor node 107 as obtained in the inference, and 2) a factor that is associated with an activation 104Vof a further operand 112 of the dot product 114, and 3) a quotient of a) a FH240201PEP-2025036018.DOCX relevance score 103lthat is associated with the successor nodes 121 as a dividend and b) a sum of I) a product of I.1) a further factor and I.2) a sum over the products of I.2.1) the activation of the predetermined predecessor node (107) as obtained in the inference and I.2.2) an even further factor that is associated with an activation 104Vthe further operand 112 of the dot product (114) and I) a number near zero or zero, such that a divisor is unequal zero, as the divisor. In Fig.1d an exemplary section of the ML predictor 101 is shown, which represents a softmax function 115 or a softmax function 115 component in the ML predictor 101 involved in the inference. Such section can, for example, be part of an attention module of the ML predictor 101 such as the attention module 122. Alternatively, the section may be outside any attention module. The softmax function 115 is applied to a matrix 113, resulting in the matrix 113’. The matrix 113 can, for example, be a set of nodes and / or the values thereof, representing, e.g., the result of the multiplication of a query matrix and a keys matrix in an attention module of the ,ொ^ML predictor 101 (e.g.ൈ^^ௗೖ) during the inference. Elements of the matrix 113’ represent successor nodes 121 and elements of the matrix 113 represent predecessor nodes 107 of the softmax function 115. In Fig. 1d, other than in Fig. 1b, the predecessor nodes 107 are the immediate predecessor nodes 107 of the successor nodes 121, while in Fig. 1b the predecessor nodes 107 relate to the predetermined nodes 107 and the successor nodes 121 relate to the further predetermined node. During back-propagating, the successor nodes 121 can have relevance scores 103l+1associated with them. Fig. 1d illustrates, as to how these relevance scores 103l+1might be back-propagated through the softmax function 115. For example, the successor nodes 121 can represent a layer l+1 and the predecessor nodes 107 can represent a layer l. The predecessor nodes 107 can, for example, be part of the portion 102 to be evaluated, or may be part of portion 123 and the successor nodes 121 can be part of the portion 123 of the ML predictor 101. Activation 104s11(^^^^^^^), 104s21(^^ଶ^^^^), 104s12(^^^^^ଶ^) and 104s22of the successor nodes 121 and activation 104x11(^^^^^^^), 104x21(^^ଶ^^^^), 104x12 , the predecessor nodes 107 are obtained during inference. The activations 104x11, 104x21, 104x12and 104x22are the inputs of the softmax function 115 and the activations 104s11, 104s21, 104s12and 104s22are the outputs of the softmax function, such that ^^^^^^^^^^^^^^^104^௫^ ^ ^ൌ104^^or equally: ^^^^^^^^^^^^^^൫^^^^^ ൯ൌ^^^^^^^. The illustration of the matrix 113 is merely representative and any dimensions for the matrix are possible, such as an n- dimensional matrix or a vector. If the input of the softmax function is a vector the formulas result in: ^^^^^^^^^^^^^^^104^௫^ൌ104 ^ ^ or equally: ^^^^^^^^^^^^^^൫^^^^൯ൌ^^^^^^. It is also mentioned, that a single index can indicate a combination of two indices. In other words, the softmax function 115 maps each of the predecessor nodes 107 onto one of the successor nodes, namely a FH240201PEP-2025036018.DOCX corresponding successor node, during inference. The softmax depicts a non-linear non- bijective function and its inverse is non-trivial. Accordingly, in back-propagating the relevance score 103 to a predetermined predecessor node not only the relevance score of the corresponding successor node is taken into account, but also the relevance scores of the other successor nodes. Additionally, the activation of the predetermined predecessor node as well as the activation of the corresponding successor node are utilized during back-propagating. The apparatus 100 can be configured to, in back-propagating the relevance scores 103l+1, back-propagate the relevance scores 103l+1at successor nodes 121 of the softmax function 115 onto a predetermined predecessor node (e.g., any of the predecessor nodes 107) of the softmax function 115, wherein a relevance score 103 of the predetermined predecessor node is calculated using ^^^൫^^^^^^-^^^∑^wherein ^^^denotes the softmax function’s 115 input for the predetermined predecessor node 107 as obtained in the inference; ^^^^^^denotes the relevance score 103l+1iof the corresponding successor node 121;∑^^^^^^^denotes a summation over the relevance scores 103l+1at the successor nodes 121 of the softmax function 115; and ^^^denotes the softmax function`s (115) output for the corresponding successor node 121. For example, if the predetermined predecessor node has the activation xl11, then softmax function’s 115 input for the predetermined predecessor node as obtained in the inference is xl11.Then the softmax function’s 115 output for the corresponding successor node is sl+111and the corresponding successor node is associated with the relevance score Rl+111. For example, for the predetermined node with the index 11, it would result in ^^^^^ ൌ In other words, the apparatus 100 can also be configured to, in back-propagating the relevance scores 103l+1, back-propagate the relevance scores 103l+1at successor nodes 121 of a softmax function 115 (e.g., of a predetermined attention module) onto a predetermined predecessor node (e.g., one of the predecessor nodes 107) of the softmax function 115, by forming a product between 1) a difference between a) a summation over I) the relevance scores 103l+1at the successor nodes 121 of the softmax function 115, multiplied with the softmax function`s 115 output for a corresponding successor node 121 associated with the predetermined predecessor node, as obtained in the inference, and b) the relevance score 103 of the corresponding successor node, and 2) the softmax function`s 115 input for the predetermined predecessor node. The back-propagation rules as they are outlined here for the softmax function 115 can, for example, be also applicable for other functions, that resemble a softmax function or encompass a softmax function or a function, that the softmax function can be derived from. FH240201PEP-2025036018.DOCX In Fig. 1e an exemplary section of the ML predictor 101 is shown, which represents a normalization function 117 application in the ML predictor 101 involved in the inference. Such section can, for example, be part of an attention module of the ML predictor 101 such as the attention module 122. Alternatively, the section may be outside any attention module. The normalization function 117 is applied on a node 118^^-^, and results in the node 118^^. The node 118^^ has a relevance score 103^^ associated with it. The apparatus 100 can, according to embodiments, back-propagate the relevance score 103^^ by setting the relevance score of the predecessor node 118^^-^of the successor node 118^^ to the same relevance score 103^^. For example, ^^^^-^ൌ ^^^^. The node 118^^-^can, for example, be part of the portion 102 to be evaluated of the ML predictor 101 and the node 118^^ can, for example, be part of the portion 123. The normalization function can be, for example, LayerNorm (Ba et al. 2016) (equation (16)), RMSNorm (Zhang and Sennrich, 2019) (equation (17)) or any function, that follows equation (18). The apparatus 100 can also be configured to, in back-propagating the relevance scores 103, back-propagate only relevance scores 103 with an absolute value greater than a certain threshold and / or with greater than zero. The portion 102 to be evaluated of the ML predictor 101 can also be an input or an input layer, wherein the input or input layer can represent anything that is given to the ML predictor 101, such as a prompt or an image. The apparatus can assign relevance scores to input nodes, which, for example, can be directly analyzed and understood by a human user. The apparatus from Fig.1a can, for example, individually operate according to one of the Fig. 1b-e or operate according to a subset of the Fig.1b-e or according to all of the Fig.1b-e. In the following, various embodiments for aspects of the present application are described which make use of the concept described above for apparatus 100 with respect to Fig.1a-1e. All details described above may be, individually or in combination, be used to extend the description and functionalities and properties of the embodiments described now below. Fig. 11 shows a schematic view of an apparatus 1120 according to a further aspect of the invention where the relevance scores are used to optimize an ML predictor. A ML predictor 101 (e.g., an LLM or ViT) is shown. The apparatus 1120 comprises an apparatus 100 and is configured to use the relevance scores 103, obtained by the apparatus 100 to control pruning and / or quantizing of the ML predictor 101. The pruning and / or quantizing yields a ML predictor FH240201PEP-2025036018.DOCX 101’. The ML predictor 101 may be an LLM, or a ViT, or any other sort of ML predictor, and the ML predictor 101’ may be, as a result of the pruning and / or quantization, be, relative to the ML predictor 101, specialized in a certain proper subset of tasks (e.g., for a set of queries, e.g., the set of queries relating to a specific subject) for which the ML predictor 101 is initially dedicated. Imagine, for example, the back-propagation of apparatus 100 is applied to obtain average relevance scores obtained by a plurality of inferences performed by the ML predictor 101 based on a plurality of inputs with then using the resulting relevance scores for pruning parts of the ML predictor 101 whose relevance scores are below a certain threshold or the like and / or quantizing weights of the ML predictor 101 according to the relevance scores to render the ML predictor 101’ sparser. For example, the plurality of inputs could comprise a context input for the ML predictor 101 (e.g., an LLM). In other words, Fig. 11 shows the apparatus 1120 for pruning and / or quantizing the ML predictor 101 (e.g., an LLM or ViT) including the one or more attention modules 122 and / or the softmax function 115. To this end, the apparatus 1120 comprises an apparatus 100 or, in other words, is configured to determine relevance scores 103 by back-propagating one or more initial relevance scores 103 at the output 110 of the ML predictor 101 or at an internal portion of the ML predictor 101, on the basis of activations 104 of the ML predictor 101, which manifest itself in one or more inferences performed by the ML predictor 101 upon being fed with one or more inputs (e.g., queries / prompts, that for example all relate to a specific domain) in portions 123 of the ML predictor 101, which lie upstream relative to the output 110 or the internal portion along an activation direction 105 of the ML predictor 101, and include the one or more attention modules 122 and / or the softmax function 115 The apparatus 1120 can be configured to, using the relevance scores 103, control a pruning and / or quantizing of the ML predictor (103). Fig.12a shows a schematic view of an apparatus 1230 according to an aspect of the invention wherein the relevance score determination is performed to determine a relevance of a context input. A ML predictor 101 (e.g., an LLM or ViT) is shown. The ML predictor is configured to be fed with a query 1232 along with context inputs 12311, 12312and 12313and so as to generate an output 110. The apparatus 1230 can be configured to determine relevance scores 103’1,103’2and 103’3, that are indicative of a relevance of the context inputs 12311, 12312and 12313for the output 110 of the ML predictor 101. For instance, the ML predictor 101 can, for example, be an LLM, the query 1232 can represent a prompt provided to the ML predictor 101 such as by a user for instance. The prompt can, for example, also be a sequence of tokens. The prompt can, for example, also be an encoded representation of a picture, an audio file, or a video. The context inputs 1231 can, for example, be a database, such as the database of Wikipedia and FH240201PEP-2025036018.DOCX the individual context inputs 1231 can, for example, be the individual Wikipedia articles or entries or parts thereof. In other words, Fig. 12a shows the apparatus 1230 for assigning relevance scores 103 to context inputs 1231 of a ML predictor 101 (e.g., an LLM or ViT) including one or more attention modules and / or a softmax function, the relevance scores 103 being indicative of a relevance of the context inputs 1231 for an output of the ML predictor 101 output 110 by the ML predictor 101 when fed with a query 1232 along with the context inputs 1231. To this end, the apparatus 1230 comprises an apparatus 100 or, in other words, is configured to determine the relevance scores 103 (e.g., to portions to be evaluated) by back-propagating an initial relevance score 103 at the output 110 of the ML predictor 101 or at an internal portion of the ML predictor 101, on the basis of activations 104 of the ML predictor 101, which manifest itself in an inference performed by the ML predictor 101 upon being fed with the query along with the context inputs 1231 in portions 123 of the ML predictor 101, which lie upstream relative to the output 110 or the internal portion along an activation direction 105 of the ML predictor 101, and include the one or more attention modules 122 and / or the softmax function 115. The apparatus 1230 can, for instance, be configured to perform a highlighting of one or more portions in the context inputs 12311,12312and 12313depending on the relevance scores 103: 103’1, 103’2and 103’3. For instance, apparatus 1230 may be configured to present, e.g., by a user interface such as a display or the like, a graphical user interface to the user that visualizes the context inputs 1231 and highlights one or more portions thereof, corresponding to the individual portions of the context inputs or parts thereof. The highlighting may be used for those context inputs 12311, 12312and 12313whose relevance scores for the output 110 turns out to be higher than a predetermined threshold and may be done by use different colors compared to those context inputs which turned out to be of no, or not sufficient, relevance. Other forms of highlighting might be used as well. The highlighting can, for example, be useful in genome analysis, where an ML predictor 101 derives its output based on a genome whose sequences are provided as context inputs 1231. This can drastically simplify genome research, as the highlighting can help identify significant positions in the genome sequence. Alternatively, the apparatus 1230 can be configured to present, to a user, via a user interface such as screen or the like, a selection or subset of the context inputs 1231 depending on the relevance scores 103. That is, the selection or subset may be determined such that the selection or subset merely comprises those context inputs 12311, 12312and 12313whose FH240201PEP-2025036018.DOCX relevance scores for the output 110 turns out to be higher than a predetermined threshold. For instance, each of the context inputs 1231 may be a file and the apparatus 1230 may be configured to automatically open and present to the user present those files, appearing to be relevant to the query 1232 based on the assigned relevance scores 103. Unprocessed context inputs mapped onto the context inputs 1231 via precoding can, for example, be a latent space representation of further context inputs which, for example, can be documents or a database. The apparatus 1230 can, for instance be configured to generate a data structure which associates a specific part of an output signal at the output 110 of the ML predictor 101 with a specific part of the context inputs 1231, or to present to a user association hints which associate a specific part of the output 110 of the ML predictor 101 with a specific part of the context inputs 1231. That is, in this variant, the relevance score determination is run from the output signal completely towards the context inputs 1231 and the determination is done in units of one or more parts of the context inputs including the specific part. By this measure, the attribution of relevant context inputs to the output signal may be made on a per part basis. That is, different context inputs 1231 may be appointed relevant context inputs 1231 for different parts of the output signal output at output 110. For example, the apparatus 1230 can be configured to present the user the outcome of the inference, i.e., the output signal, accompanied with links to the specific parts of the context inputs. For example, the apparatus 1230 could link to a specific passage of a document, that contains the information and is one of the context inputs, to a specific part of the output signal, for which the ML predictor 101 used the context inputs 1231 a context for a query to which the output signal represents an answer of the like. This can also be very helpful for summarizing documents and videos, where, for example, the specific part of the output could be a text passage and the specific part of the context inputs 1231 can be a short section of the video to be summarized. For example, the apparatus 1230 is able to provide relevant document names or link to specific sections in a document, where the ML predictor 101 retrieved its information from. For example, the mentioned examples could be realized with a Pop-up window, that displays the specific part of the context inputs 1231. The association hint can, for example, be a citation of the specific part or the presentation of the specific part of the context inputs 1231, or the specific part of unprocessed context inputs mapped onto the context inputs 1231 via precoding is a quotation out of the context inputs 1231. The apparatus 1230 can, for example, be configured to be used as an automatic citation helper, wherein a ML predictor 101 is generating text as an output 110 and the apparatus 1230 the citations are guiding a reader to the relevant text passages. FH240201PEP-2025036018.DOCX The apparatus 1230 can, for instance, be configured to sort the context inputs 12311, 12312and 12313depending on the relevance scores 103’1, 103’2and 103’3. For instance, apparatus 1230 may be configured to present, via a user interface such as a user interface, a graphical user interface to the user that visualizes the context inputs 1231 in a sorted order chosen depending on the relevance scores 103, such as in a descending or ascending relevance order. For example, in Vision Transformers (ViT) the query can represent a user instruction and the context inputs 1231 can represent Image patches, spatial information or the like. Fig.12b extends the Fig.12a by additionally illustrating an encoder 1234, that processes the context inputs 1231 to generate a latent space representation 1233 of the context inputs 1231. In an example, the encoder 1234 is part of the ML predictor 101 and accordingly, the relevance scores 103 can be assigned up until the context inputs 1231 as in Fig.12a. The encoder 1234 can, for example, be a LLM, that is configured to summarize the context inputs 1231, which then would be the latent space representation 1233. Also, the encoder 1234 can, for example, be an encoding model, that is configured to generate latent space representations 1233 of the further context inputs 1233. For example, the latent space representation 1233 can, for example, be a vector based representation of the context inputs 1231. Also, any other processing ML predictor is possible for the encoder 1234. Also, any other ML predictor, the apparatus 100 can back-propagate through, is imaginable for the encoder 1234. However, the encoder can also be outside the ML predictor 101, in which case the encoder 1234 would not be part of the back-propagation, and the relevance score determination made depending on the relevance scores is done in latent space 1233, rather than up to the initial not pre-processed context inputs. However, there as a unique association between latents 1233 and inputs 1231 and, accordingly, the user may be provided with the indication of the aforementioned highlighting, selection or sorting in the input space 1231 by using this unique association. Thus, for instance, the highlighting, selection or sorting of latents 1233 might be turned into, via this unique association, to a corresponding highlighting, selection or sorting of corresponding inputs 1231. The inputs 1231 may be present, or may be, if not present, derived by decoding, i.e., reversal of the pre-encoding by encoder 1234, the latents 1233 and performing the highlighting, selection or sorting with respect to the inputs 1231. The decoding, if needed, may be restricted to only those latens and corresponding context inputs 1231 related to the highlighting, selection or sorting. Even further, the back-propagation may be extended FH240201PEP-2025036018.DOCX using the relevance scores 103 for the latens through the encoder 1234 if the latter is another ML predictor, in order to obtain relevance scores for the context inputs 1231. Fig.14 schematically illustrates an apparatus 1440 configured to control an activation filtering functionality 1441 of a ML predictor 101 (e.g., an LLM or ViT) including one or more attention modules 122 and / or a softmax function 115. The activation filtering functionality 1441 is controllable to dampen or emphasize the activation occurring during the inference process of one or more portions of the ML predictor 101 such as individual or collection of nodes, individual layers of the ML predictor 101 or the like. The apparatus 1440 can comprise an apparatus 100 to obtain relevance scores 103. The apparatus 1440 can be configured to using the relevance scores 103, control the activation maximization functionality 1441. For instance, using apparatus 100, relevance scores 103 are determined for individual portions of the ML predictor, and depending on the application, the activation filtering functionality 1441 of a ML predictor 101 may be controlled to either dampen or emphasize these portions depending on their relevance scores such as at a strength which depends on the relevance score assigned the respective portion. This functionality can, for example, be used to perform individual neuron control. In other words, Fig. 14 shows the apparatus 1440 for controlling the activation filtering functionality 1441 of the ML predictor 101 (e.g., an LLM or ViT) including one or more attention modules 122 and / or a softmax function 115. To this end, the apparatus 1440 comprises an apparatus 100 or, in other words, is configured to determine relevance scores 103 (e.g., to portions to be evaluated) by back-propagating an initial relevance score 103 at the output 110 of the ML predictor 101 or at an internal portion of the ML predictor 101, on the basis of activations 104 of the ML predictor 101, which manifest itself in a prior (or in other words previous) inference performed by the ML predictor 101 in portions 123 of the ML predictor 101, which lie upstream relative to the output 110 or the internal portion along an activation direction 105 of the ML predictor 101, and include the one or more attention modules 122 and / or the softmax function 115. The apparatus 1440 can be configured to, using the relevance scores 103, control the activation filtering functionality 1441. An apparatus can be imagined, that, applies ActivationMaximization (ActMax) or RelevanceMaximization (RelMax) during an inference of a ML predictor, to influence the output of the ML predictor 101, based on relevance scores, obtained using the apparatus 100. Fig.15 schematically illustrates an apparatus 1550, that, during an inference of a ML predictor 101 (e.g., an LLM or ViT) including one or more attention modules 122 and / or a softmax FH240201PEP-2025036018.DOCX function 115, influences an output 110 of the ML predictor 101, by emphasizing or dampening (e.g., enhancing or suppressing) an activation of a specific neuron 1551 (e.g., selected using ActMax or RelMax ) during the inference, the specific neuron 1551 being selected based on relevance scores103, obtained by an apparatus 100 during one or more previous inferences of the ML predictor 101. The apparatus 1550 can, for example, be used to suppress unnecessary knowledge of the LLM, such as for a ML predictor, that is configured to design mechanical structures based on physical principles an activation of the specific neuron 1551, that has shown to have a high relevance score for something off topic, like biology, can be suppressed. This can lead to less energy consumption during inference of the ML predictor as well as more accurate results. Fig.16 schematically illustrates an apparatus 1660 for providing a certainty measure 1661 for an output 110 (in other words, output signal) of a ML predictor 101 (e.g., an LLM or ViT) including one or more attention modules 122 and / or a softmax function 115, comprising an apparatus 100, that obtains the certainty measure 1661 based on (e.g., a distribution) of the relevance scores 103. In other words, Fig.16 shows the apparatus 1660 for providing the certainty measure for the output 110 of the ML predictor 101 including one or more attention modules 122 and / or a softmax function 115. To this end, the apparatus 1660 comprises an apparatus 100 or, in other words, is configured to determine relevance scores 103 (e.g., to portions to be evaluated) by back-propagating an initial relevance score 103 at the output 110 of the ML predictor 101 or at an internal portion of the ML predictor 101, on the basis of activations 104 of the ML predictor 101, which manifest itself in an inference performed by the ML predictor 101 in portions 123 of the ML predictor 101, which lie upstream relative to the output 110 or the internal portion along an activation direction 105 of the ML predictor 101, and include the one or more attention modules 122 and / or the softmax function 115. The apparatus 1660 can be configured to, using the relevance scores 103, obtain the certainty measure 1661. The certainty measure 1661 can, for example, be designed to indicate how reliable the output 110 is such as, for instance, how reliable the output 110 is with same representing an answer to a query input into the ML predictor 101) for the inference based on which apparatus 100 determines the relevance scores 103. The determination may, for example, depend on a ratio between the sum of relevance scores 103 assigned to context inputs 1231 on the one hand to the sum of relevance scores 103) assigned to the query 1232 or the sum of relevance scores assigned to the query 1232 and the context inputs 1231 on the other hand. For example, the FH240201PEP-2025036018.DOCX certainty measure 1661 may be determined in a manner so as to monotonically increase with the sum of relevance scores 103 assigned to context inputs 1231 (e.g., or for specific parts of the context inputs 1231). The certainty measure 1661 may alternatively be determined to measure the proportion of relevance scores 103 which is not fully back-propagated through the network (e.g., ML predictor 101) due to the suppression of further back-propagation of relevance scores 103 of nodes which are below a certain threshold as described for certain embodiments above. The certainty measure 1661 can, for example, be indicative whether the ML predictor 101 is hallucinating. For example, if most of the relevance score is assigned to a credible source in the context inputs, the ML predictor 101 can be trusted. Additionally, we can e.g., identify attention heads or neurons that are indicative of specific behaviors of the model (e.g., ML predictor 101), such as when the model exhibits uncertainty or struggles with conflicting evidence. Fig. 17 schematically illustrates an apparatus 1770 for auditing a ML predictor 101 (e.g., an LLM or ViT) including one or more attention modules and / or a softmax function. An output 110’ of the ML predictor 101 is composed of a plurality of portions associated with a plurality of tokens. The apparatus 1770 can be configured to assign relevance scores 103 to portions 102 to be evaluated of a machine learning (ML) predictor 101 for each of a subset of the plurality of portions by back-propagating an initial relevance score 103 at the respective portion, on the basis of activations 104 of the ML predictor 101, which manifest itself in an inference performed by the ML predictor 101 in portions 123 of the ML predictor 101, which lie upstream relative to the output 110’ along an activation direction 105 of the ML predictor 101, and include the one or more attention modules 122 and / or the softmax function 115, and select among the plurality of tokens based on the relevance scores 103 obtained for the subset of portions and the output 110’ of the ML predictor 101. For example, the selected token can be provided at an output 110’’ of the apparatus 1770. The plurality of portions of the output of the ML predictor 101 can, for example, be neurons of an output layer of the ML predictor, that can be associated with a prediction of the ML predictor 101 of a specific token. The plurality of portions can also be associated with probabilities for the plurality of tokens. The portions to be evaluated can be neurons (e.g., complete layers) of the ML predictor 101, that for example relate to, inter alia, a query or context input that were given to the ML predictor 101 at input neurons, or also internal portions of the ML predictor 101. The subset of the plurality of portions can, for example, be one or more individual portions of the plurality of portions that are back propagated separately or all of them. It is possible to FH240201PEP-2025036018.DOCX select the subset based on the portions of the output 110’ of the ML predictor 101. For example, the apparatus 1770 can be configured to select those portions out of the plurality of portions of the output 110’ which are associated with most probable tokens so that the subset contains the portions corresponding to the most probable tokens. The selection among the plurality of tokens can, for example, be based solely on the relevance scores 103 obtained for the subset of portions or a distribution of the relevance scores or in combination with the plurality of portions, e.g., the activations at an output layer of the ML predictor 101. The apparatus 1770 can, for example, obtain the final token to be derived from the output 110 based on a distribution of the relevance scores 103 of at least two specific portions. For example, if for a first portion a large proportion of the relevance scores gets assigned to an input prompt (e.g., query) of an LLM this first portion and its corresponding token might be a worse prediction of the ML predictor 101, compared to a second portion, for which more relevance gets assigned to a database (e.g., a context input), that can be provided to the LLM. Depending on the embodiment and the specific intended use of the ML predictor 101 it might be the other way round or other criteria can be considered. In this description, unless specified otherwise, an inference of a ML predictor 101 can, for example, relate to an inference of a single token or an inference of an output 110 in an auto- regressive manner. The output 110 for the inference of a single token can, for example, be the output 110 of a plurality of portions relating to a plurality of tokens, e.g., activations or probabilities for the corresponding tokens, wherein sampling can be performed upon, or also an already selected single token, e.g., a token whose corresponding portion exhibits the highest value. When specifying the inference as the inference of single token (or also if the output 110 is a plurality of portions), the apparatus 100 may, for example, only need to back- propagate the once through the ML predictor 101 to assign relevance scores 103. The output 110 can, for example, also be a set of tokens (e.g., the complete generated results of an ML predictor 101 (e.g., LLM) to a prompt), the apparatus 100 can be configured to assign relevance scores 103 for each token of the set of tokens individually and obtain relevance scores 103 for the set of tokens based on the relevance scores 103 assigned for each token, e.g., by averaging or summation. In other words, in this description, the output 110 of a ML predictor 101 can, for example, be a specific token or a probability distribution over a set of tokens. The apparatus 100 can be configured to assign relevance scores 103 based on an inference leading to a token but also can be configured to assign relevance scores 103 to a sequence of tokens, e.g., the output 110 being a complete sentence, generated by a ML predictor 101. The apparatus 100 can be FH240201PEP-2025036018.DOCX configured to assign relevance scores 103 for each token of the sequence of tokens and average the relevance scores 103 to provide averages relevance scores, which can be analyzed and treated just as relevance scores 103. The generation of the output 110 may be done in an auto-regressive manner by the ML predictor 101. In this description, unless specified otherwise, the portions 102 to be evaluated can be neurons (e.g., complete layers or part of layer, e.g., complete layers without bias neurons) of the ML predictor 101, that for example relate to, inter alia, a query or context input that were given to the ML predictor 101 at input neurons, or also internal portions of the ML predictor 101. In the following, the description is resumed with more concrete embodiments. Within same, the ML predictor 101 is often exemplified as “model”, “Neural Network” or “LLM”. It should be noted that, nevertheless, the description outlined below may be extended towards other sorts of ML predictors 101 than indicated at the specific portions in the description below. Similarly, relevance score 103 is denoted below simply as relevance, attributions, relevance values and assignments, and these terms shall be treated as being synonymous. Briefly summarizing, the above described concepts make relevance back-propagation, such as LRP, applicable to a ML predictor 101 with one or more attention module 122 and / or a softmax function 115. ML predictors may be a neural networks. Examples for ML predictors with one or more attention module 122 and / or a softmax function 115 include, exemplarily but not exclusively, transformer architectures, such as discriminative and generative Large Language Models and Vision Transformers and Multimodal Models, such as the Open-Source LLama variants published by Meta AI. As shown, the above concepts include, inter alia, ones having the property that they are able to propagate relevances, called relevance scores 103 in the above description, through a SoftMax component (e.g., softmax function 115) of an ML predictor (e.g., ML predictor 101), synonymously called softmax function 115 in the above example, and / or through an Matrix Multiplication component of an ML predictor, synonymously called dot product 114 element herein, and / or Normalization layer of an of an ML predictor (e.g., ML predictor 101), synonymously called normalization function 117 herein. Naturally, an ML predictor (e.g., ML predictor 101) of the present application needs not to comprise all of said components. An ML predictor may, among others, fall into one of the following exemplary types of (non- linear) Neural Network Modules (or in other words, Neural Network modules), namely one comprising one of the following: FH240201PEP-2025036018.DOCX - a normalization layer: Normalization Layers (e.g., Examples of normalization function 117) can, for example, be LayerNorm (Ba et al., 2016), as an instance of Normalization Layers (Equation (16)) and / or RMSNorm (Zhang and Sennrich, 2019), as an instance of Normalization Layers (Equation (17)) and / or all layers that perform a normalization operation as described in Equation (18), where g(x) is an arbitrary function as well as further types of normalization layers. - an attention module: An attention module (e.g., attention module 122) can, for example, be non-linear Attention, such as Grouped Query Attention (Ainslie et al., 2023), Multi-Query Attention (Shazeer., 2019), Global / Local Attention (Luong et al., 2015), Soft / Hard Attention (Xu et al., 2015), Cross Attention (Bahdanau et al., 2015, Vaswani et al., 2017), Self-Attention (Cheng et al., 2016, Zhang et al., 2019) as well as further types of Attention modules. An attention module can also, for example, be linear Attention. An apparatus (e.g., apparatus 100) according to an embodiment can attribute through bi-linear matrix multiplication (e.g., dot product 114). This is important for possible linear attention mechanisms. The above-mentioned modules, such as dot product 114, normalization function 117 and / or softmax function 115, may constitute building blocks of various Transformer architectures, but also other contemporary Neural Network architectures, including, the models which are considered Transformer-based models as well as networks with Attention which are NOT necessarily transformers, including – and thereby forming further examples for the above ML predictor: - Large Language Models (LLMs), (generative) Transformers in Natural Language Processing (NLP), - Vision Transformers (ViT), (generative) Transformers in Computer Vision - Cross-Domain / Multimodal Transformers. For multimodal models such as these. - Explanations as well as relevance scores can be derived for all input and output modalities at the same time - Transformer architectures in other domains, which can be provided to the model as sequences of features can include, e.g., Audio, (Medical) Time Series, Video, Collections of Documents - Recurrent Neural Networks, such as Single Headed Attention RNN (Merity, 2019), a Sequence to Sequence (seq2seq) model with Attention modules - Convolutional Neural Networks (CNNs) with Self-Attention (Yang et al., 2019) FH240201PEP-2025036018.DOCX - Mixture of Experts (MoE) models (Shazeer et al., 2017; Fedus et al., 2022). Models that use conditional computation and composed of set of experts, usually Feed Forward Neural Networks (FFNNs), and a trainable gating network. Unlike standard NNs, which usually only have a single set of parameter for inference, MoE models can use different combination of parameters to process each input. - Various Graph Neural Networks (Scarselli et al., 2009; Veličković et al., 2018). Models that take graph-structured data as input and perform operations over them: - Models that mix between attention layers and components of other architectures, where the Jamba models are an example. Jamba model (Lieber et al, 2024) is a hybrid between Mamba (Albert et al, 2023) and Transformer models. Note that Transformers can be used to generate new features (images, text) (e.g., at the output 110) based on input stimuli (e.g., given to the input 109), even across modalities. Transformers can however also be used as part of discriminative models, e.g., as feature extractors for classifiers or regression models (which both can be handled with relevance back-propagation as presented herein. Note that, as described above, the relevance scores can, for example, be computed wrt. the input (e.g., a prompt / or a question to LLM (e.g., ML predictor 101)), wrt. intermediate neurons in the model (encoding concepts) and / or wrt. context given to the model (e.g., documents in an external database). The back-propagation concepts described and presented herein are also briefly denoted as AttnLRP (Attention-LRP), and, according to certain embodiments thereof, have various key benefits which set them apart from competing / existing methods applicable to Transformer-like architectures: AttnLRP is extremely efficient, computationally, minimizing the cost to obtain explanations (e.g., through assigned relevance scores 103) for sets of, or single, generated tokens / units of information. The cost of one AttnLRP computation (e.g., one time back-propagation the initial relevance score 103ithrough the ML predictor 101) resides in the order of magnitude of a single forward pass through the model. Other approaches, based on Perturbation (e.g., AtMan (Deiseroth, 2023), SHAP(Lundberg, 2017)) can explain (i.e., compute attributions) at costs of several orders of magnitude higher (hundreds or multiple thousands of times more expensive). The obvious benefit to the stakeholder here is the economic nature of embodiments of the present inventive concept, which directly translates into expense of time, energy, CO2 and money. With AttnLRP, real-time applications fulfilling the transparency criteria of current and future AI regulations can be realized. FH240201PEP-2025036018.DOCX In terms of resource use, AttnLRP can be configured to be very fast (at the expense of memory), or more memory efficient (at the expense of computational efficiency), via Attribution Checkpointing, enabling the stakeholder to tune the approach (e.g., using an apparatus 100 according to an embodiment of this invention) to the particular requirements of the setting / situation. An apparatus (e.g., apparatus 100) according to an embodiment of this invention may or may not follow the conservation property of LRP. Hence, the outgoing relevance computation of tokens receiving low relevance values (in the output space of the model (e.g., ML predictor 101) and / or the latent space (e.g., in any hidden layer) of the model (e.g., ML predictor 101)) can be optionally skipped (e.g., if the corresponding relevance score is below a threshold) due to the sparsity of the relevance distribution; and the method can be extended to sparse backward differentiation, wherein only values with non-zero relevance are computed, leveraging the sparsity of the relevance distribution. At the same time, the explanations (e.g., derived from the assigned relevance scores 103) obtained through AttnLRP are very faithful to what the model (e.g., ML predictor 101) was doing during the prediction process (e.g., during inference). Other methods can be very noisy, or less faithful in general. Hence, the quality of the explanations (e.g., the quality of the relevance scores 103) obtained by AttnLRP exceeds those of related methods, immediately translating into a “higher quality measurement” taken based on AttnLRP attributions. AttnLRP is a modified backpropagation method. This means, after having propagated once through the model (e.g., ML predictor 101), all model components (e.g., portions 102 to be evaluated) have an associated attribution score (e.g., the assigned relevance scores 103). One feature of AttnLRP here is, that all components of a Transformer-like architecture (e.g., of an attention module of ML predictor 101) can be handled / reached, especially the Attention modules, of which other methods are incapable. This allows a “look into” the attention module (e.g., attention module 122), i.e., analyze which interactions between tokens (i.e., given input words in text, or image regions in vision) are important etc. (i.e., it allows attribution to attention subspaces / latent spaces.) The latent space attributions (e.g., the assigning of relevance scores 103 to non-input and non- output layer nodes) of AttnLRP further constitute a foundation to implement a CRP-derivative (e.g., Concept Relevance Propagation (CRP) derivate) (a potential AttnCRP), which would combine the strength of both methods, and can allow for concept-based explanations for FH240201PEP-2025036018.DOCX Transformers, with all consequences (i.e., tools and mechanisms such as relevance-guided reverse searches) The description proceeds with a more detailed presentation of embodiments for the relevance back-propagation further below. Before, however, a presentation of applications and use cases of AttnLRP is presented. Note, that the description presented below primarily deals with LLM (Transformers operating in Natural Language Processing (NLP, i.e., Text) as an example for the ML predictor. However, the same techniques and, thus, the embodiments set out herein, are applicable in other data domains, such as audio, video, images, sequences of genomes, etc, i.e with respect to other sorts of ML predictors. Even a usage across domains is possible, especially with multimodal models. One example for the application of the presented AttnLRP methods is the optimization, i.e. pruning and / or quantization, of transformer-like architectures, based on latent LRP (or AttnLRP) attributions. A derivative thereof may be a specialized Sub-Network Extraction: Consider taking a generic, pretrained and large LLM, extracting a more sparse / efficient subnetwork which is skilled in application domain X (by computing relevance scores wrt samples from domain X), but unskilled in domains not X, which are not of interest. Thus, it is possible to make an LLM much smaller, allowing it to run on smaller devices, more energy efficiently. For example, the apparatus 1120 of Fig.11 can be configured to extract a sub-network 101’ of the ML predictor 101, that is specialized on a specific domain, by providing the ML predictor 101 with samples associated with this specific domain (e.g., as an input, e.g., as prompts) and assigning relevance scores for each of the samples and pruning any node / interconnection, whose relevance scores for each of the samples are on average (e.g., or are as a sum) below a certain threshold. Another example is relevance-guided document retrieval not only based on value-path attributions in latent space, but also attention-pathway attribution (e.g., AttnLRP can reach, other methods cannot). For example, an exemplary attention module 122 may be written as^^^^^^^^^^^^^^ ൬ொ∗^^^ (e.T^ௗg., equaling softmax((Q*K ) / sqrt(dkೖ)) (e.g., see equation (10)). Everything inside the softmax function (and the output thereof) (e.g., the softmax function 115) is considered the attention pathway, while V is the value pathway. Most competing explainers for transformers do ignore the attention pathway, because it is difficult to crack. However, AttnLRP delivers a complete solution, which others do not. LLMs (e.g., the ML predictor 101) can answer questions (e.g., given to the ML predictor 101 at the query 1232) based on a document Context provided via large databases (e.g., the FH240201PEP-2025036018.DOCX context inputs 1231) (e.g., a question can be asked, and the LLM (e.g., the ML predictor 101) can be provided with one or more Wikipedia articles (e.g., as the context inputs 1321)). Through AttnLRP, one can then retrieve the (most) relevant documents, collect and present them to the stakeholder, and within those documents can “cite” the relevant sources by attributing Relevance (e.g., from the assigned relevance scores 103 of the context inputs 1231) to the corresponding words / sentences / paragraphs. This is highly relevant to stakeholders frequently (re)searching in / through data. documents could be text, video, audio, genome data, time series of any sort, large collection of stuff, graphs. anything. more. pdfs. Also, patent lawyers and patent examiner could benefit. One particular development tied to LLMs (e.g., ML predictor 101) is the introduction of RAGs (Retrieval Augmented Generation (Lewis et al., 2020)). In the RAG approach, external documents are compressed into vectors and saved in a database. This system can be independent from the LLM (e.g., ML predictor 101, e.g., can also be ViT with picture / video database compressed into vectors), where during a query search, the vectors most similar to the query are identified, and their linked context in plain text is then fed as input to the LLM. This provides the model with additional information, enhancing its predictions by leveraging relevant external data. Alternatively, there is a variant where the model and RAG system are trained end-to-end. In this setup, the LLM does not receive the document context as “plain text”, as e.g., human readable articles (or full videos), but instead as already encoded documents in vector space (e.g., latent space representation 1233), which can be directly attached to hidden / latent representations within the LLM. This yields a computationally more efficient and more memory efficient (LLMs do not need to encode the text through millions of operations, or hold all intermediate steps in memory, but have access to already pre-encoded representations of the text) predictions. Consider now one has a database linking the encoded documents (e.g., one or more of the context inputs 1231) to the original, uncompressed texts. The encoded documents in plain text or in latent space are provided to the LLM (e.g., the ML predictor 101) in a RAG, then 1. retrieve the original documents from the database (e.g., one or more of the context inputs 1231), based on their relevance (e.g., the assigned relevance scores 103) 2. only for those documents (e.g., one or more of the context inputs 1231), execute all the encoding steps, forward-propagating through the network until the encoded version is recreated. 3. Given these encodings, AttnLRP can be further applied to attribute precise word / sentence / paragraph-level encodings “citations” in the source document, FH240201PEP-2025036018.DOCX illuminating the LLMs source of information based on which an answer has been generated. → relevance (e.g., the relevance scores 103) could tell us which encoded documents are informative → decode those, then propagate relevance (e.g., relevance scores 103) further to text level.→ cite within the retrieved documents the particular important information.Likewise, the capability to “cite” sources for answers in given contextual documents allows us to measure (in terms of attribution computed (e.g., the relevance scores 103) by an apparatus (e.g., the apparatus 100) according to an embodiment of this invention) whether the LLM (e.g., the ML predictor 101) is focusing on the factual text at its disposal (relevance is then high in the portions of the documents), or whether relevance is absorbed by internal components / structures of the Transformer. In the latter case, this is an indication of the LLM hallucinating answers based on “the most likely next word / sentence” Debugging or (in)validating LLM answers is now possible. In particular, citations (via relevance attribution (e.g., by using the apparatus 1230)) enables one to infer whether an output (e.g., the output 110) is based on a citable document source (e.g., one or more of the context inputs 1231), or hallucinated based on learned parameters (in that case, strength of relevance in the input would be very weak, and conservation property between generated output / token and input space would be non-conservative). Thus, it is possible to measure whether the LLM is confabulating (e.g., hallucinating). In an embodiment, an apparatus (e.g., apparatus 1770) facilitates relevance-guided sampling of predicted tokens by selecting them based on their explained input and latent relevance distributions. Typically, LLMs (e.g., ML predictor 101) predict the continuation of an input prompt in an auto-regressive manner, generating a probability distribution for all possible next token continuations at each step. Various sampling methods are used to guide the generation process, such as selecting the next token randomly or choosing the one with the highest probability. However, using AttnLRP, the relevance of the input prompt and latent space can be computed for each possible prompt continuation, allowing us to select the next tokens based on a relevance distribution criterion. Relevance-guided sampling (e.g., an apparatus 1770 for auditing a ML predictor 101) can be applied to select tokens that yield relevance on external retrieved documents (e.g., context inputs 1231), thus excluding those originating from model parameters and reducing FH240201PEP-2025036018.DOCX hallucinations in the model's output (e.g., at output 110’). Additionally, this method can enhance the model's adherence to instructions by applying it to generated tokens (e.g., to plurality of portions at the output 110 of a ML predictors 101) with high relevance to specific parts of an instruction prompt, ensuring that the generated text aligns with the user's commands. The description succeeds with a description of potential applications, that leverage the capabilities described above. One possible application of an apparatus (e.g., apparatus 1230) can enable the building of e.g., a browser extension / integration (or other app) which can summarize webpages, using the apparatus (e.g., apparatus 1230) can give sources for the LLM summary, so the truthfulness of the LLM-derived is traceable (e.g., by providing the ML predictor 101 with the webpage as a context input 1231). Measuring whether a generated token has been predicted based on grounded (contextual) information, or based on learned (parametric) knowledge. Via relevance (e.g., relevance scores 103), the functionality of each internal “head” (attention or value or ...) of the model (e.g., ML predictor 101) can be distinguished, whether it acts on context (e.g., the context inputs 1230) or parameters, and what kind of focus (topics, semantics, etc.) they have. (This is a more general capability of AttnLRP) Another possible application of an apparatus (e.g., apparatus 1230) according to embodiments can enable LLMs (e.g., the ML predictor 101) to work scientifically and to cite its sources (e.g., sources provided as context inputs 1231), by adding the AttnLRP module to it (e.g., by utilizing the apparatus 1230 or the apparatus 100). This will become increasingly interesting, the larger the resources are available to the model (e.g., the ML predictor) (as human manual labor is slow). After giving a summary, the human can be guided to the relevant sources to fact- check / read the original (e.g., by highlighting one or more portion in the context inputs 1231 depending on the relevance scores 103’). Similar for multimodal models, analogously to text summarization, one could do "other media summarization", either in-domain, or for human more naturally digestible from <media domain> to text. One other example could be the generation of movie summaries, where AttnLRP can be used to "cite" scenes (e.g., the context inputs 1231) where the model (e.g., the ML predictor 101) gets its descriptions (e.g., the output 109) from. FH240201PEP-2025036018.DOCX A potential application for this might be keyframe collection matching the (in, with multimodal Transformers, simultaneously generated) textual description (e.g., at the output 109) of the media document (e.g., as the context input 1230). This is relevant, for example, for stakeholders with too little time to watch movies, streaming platforms, law enforcement (hours of security camera footage) and / or audio to text transcription workers (e.g., historians / scientists). Given the LLM is confronted with ambiguous information, or providing ambiguous answers (“Person X is y or z years old”). Immediately, AttnLRP attributions (e.g., relevance scores 103) can be used to measure the factual source in context for each of the answer’s options (e.g., for multiple answers (e.g., at the output 110) of the ML predictor 101 asses the relevances of portions of the context inputs and check, if they roughly align with another). Alternatively, in reaction to the identification of the ambiguous sources, one could modify the ambiguities of the context slightly, i.e., willingly create contradicting statements in the text that the LLM (e.g., the ML predictor 101) sees. If no additional useful information is provided, these statements should be returned as equally likely answers. Then, text could be added, that suggests one of the statements is likely to be true which then should shift the probability assignment to the answers. The shift in output probabilities (e.g., at the output 110) (for generated words) would then reflect in a shift in attributions at document level, i.e., through AttnLRP. This has already been mentioned above, as one of the technical capabilities we obtain through AttnLRP: Relevance-Guided on-demand RAG-decompression (summarized: use relevance information to surgically invest compute power to fully compute only informative documents, to enable citation capability through AttnLRP as a consequence (e.g., the apparatus 100 only continues assigning relevance scores during back-propagation if, at an encoding layer, a corresponding relevance score 103 is larger than a threshold (e.g., larger than zero or ) otherwise the corresponding relevance 103 score is skipped and not back-propagated further; the encoding layer can be part of the ML predictor 101 or can also be given to the ML predictor as a context input 1230). 1. Assume a huge context of encoded documents (i.e., all of Wikipedia) being available as compressed / encoded states in latent space (e.g., given to the ML predictor as context inputs 1230). 2. Ask a question to the LLM (e.g., the ML predictor 101), receive answer (e.g., output 110) 3. perform AttnLRP, select most relevant document based on attribution (e.g., on or more of the context inputs 1231 based on their assigned relevance scores 103’). FH240201PEP-2025036018.DOCX 4. Retrieve those documents, fully compute encodings for these documents from input to latent space to recreate RAG’s latent encodings. 5. Follow those compute paths backwards to the input again with AttnLRP (e.g., using the apparatus 100) (however, only for the small subset of documents (e.g., the context inputs 1230) which are informative (e.g., whose relevance score is above a threshold or above zero), which makes this efficient). 6. This will not be possible / feasible to do brute force for all documents in the context (e.g., all of the context inputs 1231) (to save compute is also the main reason RAG exists) Same as for RAG, this can be done with a chain of LLMs. Imagine two LLMs, where the first is a document preprocessor (e.g., summarizes documents) (e.g., a further ML predictor generating context inputs 1231 for the ML predictor 101 by summarizing the documents), and the second one (e.g., the ML predictor 101) then answers user questions (e.g., given as the query to the ML predictor 101) about the summarized document. With AttnLRP, one can not only explain the source of the answer in the compressed document (e.g., the output of the further ML predictor), as used by the LLM (e.g., the ML predictor 101), but one can further propagate back through the compressing LLM (e.g., the further ML predictor) to cite the answer in the original, uncompressed document. Another potential application would be the extraction of more efficient, smaller, faster, more specialized subnetworks from generic LLMs (e.g., from the ML predictor 101 using the apparatus 1220) (or other transformers), based pruning or quantization via relevance (e.g., relevance scores 103) from AttnLRP. LLMs (e.g., ML predictor 101) are orders of magnitude larger and more costly to apply than “traditional” CNNs / DNNs. With AttnLRP, one can assign relevances (e.g., relevance scores 103) to the latent parts (e.g., the operand 111, the further operand 112, the matrix 113, the predecessor nodes 107, the further predetermined node, …) of the Transformer and based on this quantity decide which parts of the model (e.g., the ML predictor 103) are functional, and which ones are irrelevant for a task. Unnecessary embedding subspaces can then be removed (e.g., using the apparatus 1220), quantized or compressed (iteratively, until a loss in performance is detected). For example, if a stakeholder wishes to run a customer support LLM (i.e., chat bot) (e.g., the mL predictor 101) which should only be knowledgeable about certain products and guidelines for the AGBs are needed (which can be provided as context for the LLM) (and not, for example unrelated world knowledge such as quantum physics), and one does want to avoid the investment of energy (and money) when running the LLM (e.g., the ML predictor 101) (related to unrelated world FH240201PEP-2025036018.DOCX knowledge), which would also execute computations in latent spaces which embody knowledge about irrelevant topics, such as here, e.g., chemistry, quantum physics, etc. 1. Sub-case, relevance-guided quantization of LLMs, i.e., not only “which filters / subspaces should be removed”. Pruning of substructures is equal to a quantization to 0 (i.e., removal of neurons or weighted connections). Instead, a quantization can be performed (e.g., by the apparatus 1220), e.g., encoding weighted connections and parameters, or even the numerical resolution of neurons (e.g., nodes) with fewer bits (float16 instead of float32 precision or 8bit integer precision etc.). 2. With the AttnLRP derived relevance quantities (e.g., relevance scores 103), above mentioned quantization can be used to guide the precision of “nuance” with which a Transformer (e.g., the ML predictor 101) is able to encode various aspects of its reasoning. 3. Another variant could be targeted at savings in either space, by minimizing the parameters of a model (e.g., ML predictor 101) with the objective of storing the model (e.g., on a mobile / edge device) or transmitting the model (over a network connection) 4. Another variant could be targeted specifically to lower bit depth of parameters (e.g., parameters of the ML predictor 101) to enable a more efficient computation. 5. Another variant could be targeted at reducing the number of parameters (e.g., parameters of the ML predictor 101) one needs to compute over, in order to enable a more efficient computation 6. A combination of all of the above in order to transfer LLMs (e.g., the ML predictor 101) to specialized compute hardware supporting e.g., only integer quantization, which is much more efficient than floating point computation. A common way to interact with a generative (LLM) Transformer (e.g., ML predictor 101) is to re-formulate the prompt given to the model with instructions on how to modify the answers given (e.g., at the output 110). This, however, leads to uncertainty on the updated answer. When modifying a prompt (e.g., given as the query 1232 or as context input 1231 or at any other input of the ML predictor 101), one must hope that the model (e.g., the ML predictor 101) reacts in the intended way, which often does not happen, and a repetition of the process is often required in order to obtain satisfactory results. With an AttnLRP-based identification of functional subunits, as well as the identification of their functionality via ActivationMaximization or RelevanceMaximization (e.g., with ActMax or RelMax it is possible to select / generate examples for words / sentences / images which a neuron of interest generally activates for or produces) can enable a more direct interaction with the FH240201PEP-2025036018.DOCX model. With AttnLRP, one can find out the most relevant latent units in the LLM / Transformer which are responsible for the generation of the outcome. The following can relate to embodiments of the apparatus 1550. The proposed concept can be called individual neuron controlling. Together with RelMax / ActMax / (or another language model (e.g., ML predictor 101) purposed solely for explaining the most relevant units (similar to OpenAI work) conditioned on the units detected by AttnLRP), the user is informed about the semantics of the relevant neurons. Then, with the information WHICH units are responsible for WHAT kind / nuance of the generated answer, PRECISELY those neurons (e.g., the specific neuron 1551) can be controlled (called neuron or neural steering). Those neurons (e.g., the specific neuron 1551) can be activated in a direct manner more / less to control particular aspects of the output generation (e.g., the output 110). This is analogous to have a dashboard with knobs and dials at one’s disposal, to deterministically gain control over the net. Furthermore, once the links between semantics and particular units have been established once, this knowledge can always be re-used leading to a very efficient and deterministic control tool over Transformer-generated content. A use case for this presented case of individual neuron controlling can be a training-free tuning of LLMs to better fit an application domain. Training LLMs (e.g., ML predictor 101) is extremely costly, requiring large amounts of data, to do so effectively. Enabling the stakeholder to tune the model (e.g., ML predictor) in an ad-hoc fashion, while avoiding training, will be very beneficial to stakeholders aiming at the deployment of (generative) Transformers (e.g., ML predictor 101). Step 0: download a foundation model (e.g., ML predictor 101) Step 1: identify certain functional neurons (e.g., the specific neuron 1551) using AttnLRP and RelMax / ActMax / etc. Examples would be neurons tied to different topics (mathematics, hate speech, structure / quality of language, complexity of language) Step 2: (de)activate those neurons to tune the quality of generated answers. (e.g., by suppressing or enhancing the activation of the specific neuron 1551) One exemplary use case can be the tuning of the output (e.g., 110) of an LLM (e.g., ML predictor 101) to e.g., "simple English" for school children in grades 1- 4. Another related use case for individual neuron controlling can be the ability to tune the model (e.g., ML predictor 101) in terms of fairness, by removing biases (permanently, by globally tuning out neurons (e.g., the specific neuron 1551), or dependent on the context (only if the FH240201PEP-2025036018.DOCX model (e.g., ML predictor 101) “uses” the racism neuron (which is measurable in AttnLRP- computed relevance quantities), alter the model here and there by suppressing neurons or sets thereof.) Another related use case for individual neuron controlling can be, that via the identification of learned “false facts” from poor quality training data with AttnLRP one can “fix” the model (e.g., ML predictor 101) by severing learned failure-inducing connections responsible for re- occurring falsehoods in the generated answers. a) This works without access to the training data the model (e.g., ML predictor 101) has seen, when this information is embedded in the model (e.g., ML predictor 101). AttnLRP can identify those structures. b) Even if we had access to the training data, each re-training would result in high monetary investments. With the AttnLRP-based identification, and then the neural tuning approach (e.g., the individual neuron controlling), these costs can be avoided largely. c) It is thinkable that above approaches could be used to increase truthfulness in LLMs (e.g., ML predictor 101) to an extent, by enforcing “truthful” neurons (e.g., the specific  neuron  1551), forcing the LLMs (e.g.,  ML  predictor  101)  to solely rely on factualinformation provided via the context. Another related use case for individual neuron controlling can be, that given an LLM / Transformer (e.g., ML predictor 101) which has been trained on largely by humans unexplored (because of the size) bodies of data, AttnLRP can be used with its capabilities of document retrieval to aid the process of scientific discovery. In an industry context, AttnLRP would operate as a tool to automate data annotation, avoiding costly and error-prone human labor. Here, one would identify important neurons (e.g., the specific neuron 1551) for <some category or sentiment or quality>, and then perform CRP or RelMax through / for those neurons in order to select and annotate (parts of) documents relevant for <some category or sentiment or quality> embodied by the currently observed / catalogued / known neuron. Another possible application of an apparatus 1230 can be the following. LLMs (e.g., ML predictor 101) can be configured to use functional APIs, by reading natural language instructions (e.g., at a query 1232 or at context inputs 1231) such as “what is the sum of three and five” or “what is the weather tomorrow” or “what is the product of the numbers in the following list: [1,24,5]”, e.g., by being given access (e.g., via example or other means) to API signatures of e.g., a calculator. The LLMs (e.g., ML predictor 101) here transfer the natural language instruction “what is the sum of three and five" into a function call signature, e.g., “sum FH240201PEP-2025036018.DOCX (3,5)”. With AttnLRP, we can ensure that the translation from the API calls are properly translated, i.e., here, call of “sum” function, translation of “three” to “3" and “five” to “5” for the function API input. For example, with the query: “What is 4 plus 5” The apparatus 1230 provide a data structure which associates a specific part of the output 110 of the ML predictor 101 with a specific part of the query 1232. For example, a data structure linking to the “4” and “5”, which then can be given to a summation API to check the results the ML predictor 101 originally provided. The ML predictor 101 can also be configured to call the API on its own. Hence, AttnLRP enables transparency on how the LLM (e.g., ML predictor 101) interacts with the API interface. Another use case would be the application of AttnLRP in order to explain multimodal Question Answering tasks, where a Transformer is given a document (pdf document, image, video) and an instruction / question at the same time. The answer can be explained towards all input sources at the same time, e.g., explaining where in an image the model has found evidence supporting the answer given by the model, and how the answer in turn relates to the question asked. The description proceeds with a more detailed explanation of the concept for relevance back- propagation as used by the above embodiments wherein this description starts with a brief summary of some main thoughts underlying these concepts. Large Language Models (e.g., ML predictor 101) are prone to biased predictions and hallucinations, underlining the paramount importance of understanding their model-internal reasoning process. However, achieving faithful attributions (e.g., relevance score 103) for the entirety of a black-box transformer model and maintaining computational efficiency is an unsolved challenge. Embodiments are extending the Layer-wise Relevance Propagation attribution method to handle attention layers, these challenges are addressed effectively. While partial solutions exist, embodiments of the present inventive concept are the first to faithfully and holistically attribute not only input but also latent representations of transformer models with the computational efficiency similar to a singular backward pass. Through extensive evaluations against existing methods on LLama 2, Flan-T5 and the Vision Transformer architecture (these are examples for e.g., the ML predictor 101), it is demonstrated that embodiments of the present inventive concept surpass alternative methods in terms of faithfulness and enables the understanding of latent representations, opening up the door for concept-based explanations. The concepts presented in the present application are, for sake of an easier reference, called AttnLRP. AttnLRP can be an extension of LRP within the Deep Taylor Decomposition framework (Montavon et al., 2017), with the particular requirements necessary for attributing non-linear attention accurately. AttnLRP can allow the explaining of transformer-based models FH240201PEP-2025036018.DOCX (e.g., ML predictor 101) with high faithfulness and efficiency, while also allowing attribution of latent neurons and providing insights into their role in the generation process (see Fig.3a). 1. Novel efficient and faithful LRP attribution rules for non-linear attention within the Deep Taylor Decomposition framework are presented herein. In addition, it is demonstrated how to combat noisy attributions in ViTs. 2. It is illustrated how to gain insights into a LLM generation process by explaining latent features. In the following description, first, LRP is motivated in the framework of additive explanatory models (e.g., ML predictor 101). Then, the design of new rules for non-linear operations is generalized. Finally, embodiments of the invention are discussed in more details than before, that utilize LRP in each operation of a transformer model with efficient and faithful rules. In the following, Layer-wise Relevance Propagation is discussed. Layer-wise Relevance Propagation (LRP) (Bach et al., 2015; Montavon et al., 2019) belongs to the family of additive explanatory models, which includes the well-known Shapley (Lundberg and Lee, 2017), Gradient × Input (Simonyan et al., 2014) and DeepLIFT (Shrikumar et al., 2017) methods. The underlying assumption of such models is that a function ^^^with ^^ input features ^^ can be decomposed into individual contributions of single input variables ^^^←^(called “relevances”, e.g., relevance score 103). Here, ^^^←^denotes the amount of output ^^ that is attributable to input ^^, which, when added together, equals (or is proportional to) the original function value. Mathematically, this can be written as: ^^^^^^ ^ ∑ே ^∝ ^^ ൌ ^^^^←^(1) If an input ^^ is connected to several outputs ^^, e.g., a multidimensional function ^^, the contributions of each output ^^ can be losslessly aggregating together. ^^^ൌ∑^ ^^^←^ . (2)This provides ”importance values” (e.g., relevance score 103) for the input variables (e.g., predecessor nodes 107), which reveal their direct contribution to the final prediction. Unlike other methods, LRP treats a neural network as a layered directed acyclic graph, where each neuron ^^ in layer ^^ is modeled as a function node ^^^^that can be individually decomposed according to equation (1). Beginning at the model output (e.g., output 110), the initial relevance value (e.g., relevance score 103) ^^^^ ∝ ^^^^is successively distributed to its prior network neurons (e.g., predecessor nodes 107) one layer at a time. Hence, LRP follows the flow of activations (e.g., activations 104) computed during the forward pass (e.g., in an activation direction 105) through the model (e.g., ML predictor 101) in the opposite direction, from output ^^^back to input layer ^^^. FH240201PEP-2025036018.DOCX This decomposition characteristic of LRP gives rise to the important conservation property: ^^^ି^ ൌ ∑ ^^ ^ି^ ൌ ∑ ∑ ^^ ^ି^^←^ ∑ ^ ^^ ^ ^ ^ ൌ ^ ^^^ ൌ ^^ , (3)ensuring that the sum of all constant. This property allows for meaningful attribution, as the scale of each relevance value can be related to the original function output ^^^. In the following, decomposition through linearization according to embodiments of the inventive concept is discussed. To design a faithful attribution method, the challenge lies in identifying a meaningful distribution rule ^^^←^. Possible solutions encompass all decompositions that adhere to the conservation property (3). However, for a decomposition to be considered faithful, it should approximate the characteristics of the original function as closely as possible. Embodiments of the present inventive concept, take advantage of the Deep Taylor Decomposition framework (Montavon et al., 2017) to locally linearize and decompose neural network operations into independent contributions. As a special case, we further establish the relationship between one derived rule and the Shapley Values framework. Starting with computing a first-order Taylor expansion at a reference point ^^^. For the purpose of simplifying the equation, it is assumed, that the reference point ^^^ is constant: ୠ୧ୟ^^^ೕwhere ^^ is the approximation error in Big- ^^ notation and the Jacobian J is evaluated at reference point ^^^, that is in the following omitted for brevity. The bias term represents the constant portion of the function and the approximation error that cannot be directly attributed to the input variables. The neural network function can be substituted with its first-order expansion and its proportionality can be asserted to a relevance value ^^^relevance score 103) throughmultiplication with a constant ^ Generally, embodiments of the invention can be focused on attributing the input variables (e.g., predecessor nodes 107) and hence may ignore the contribution of the bias term (Bach et al., 2015). However, it is important to note that the bias absorbs part of the relevance (e.g., relevance score 103) and that the conservation property (3) still holds if the bias is regarded FH240201PEP-2025036018.DOCX as an additional neuron without connection to the input. Alternatively, the bias term can be distributed equally among the input variables, as explained later. Comparing with Equation (1), ^^^←^can be identified as the summands and Equation (2) can be applied to the input variables (e.g., predecessor nodes 107). In addition, a stabilizing factor ^^ (usually set to 10-^) with the sign of ^^^^^^^ can be inserted to allow for the case ^^^^^^^ ൌ 0: ோೕ In the following, ^^^^^^^^^^^^^^^^^ is omitted for brevity. To benefit from GPU parallelization, this formula can be written in matrix form: ⇒^^^-^ൌ^^ ⊙ ^^் ⋅ ^^^ ⊘ ^^^^^^^^^^^where ⊙ denotes the Hadamard product and ⊘ elementwise division. This formula can, for example, be efficiently implemented in automatic differentiation libraries, such as Py-Torch (Paszke et al., 2019). Compared to a basic backward pass, AttnLRP can have additional computational complexity for the element-wise operations. In the following, it is explained, how the Multilayer Perceptron can be attributed. Commonly a Multilayer Perceptron consists of a linear layer with a (component-wise) non-linearity producing input activations for the succeeding layer(s): ^^^ൌ∑^^^^^^^^^^^^(6) ^^^ൌ^^^^^^^ (7) where ^^^^are the weight parameters and ^^ constitutes a (component-wise) non-linearity. For example, ^^^can denote the predetermined node 106 and ^^^can denote the predecessor nodes 107. In the following, the ^^- and ^^ -LRP rule words, ^^ – and ^^- rule) are explained. Linearizing linear layers (6) at any point ^^ ∈ can result in the fundamental ^^-LRP (Bach etal., 2015) rule ௭ೕ^௫^^ఌThe bias ^^^(Equation (6)) can absorb a portion of the relevance (e.g., relevance score 103). The proof is omitted for brevity. The ^^-LRP rule is employed on all linear layers, unless specified otherwise. In models (e.g., ML predictors 101) with many layers, the gradient of a linear layer (equal to the weight matrix ^^) can cause noisy attributions due to the gradient shattering effect (Balduzzi FH240201PEP-2025036018.DOCX et al., 2017). To mitigate this noise, it is best practice to use the ^^-LRP rule (Montavon et al., 2019), an extension to improve the signal-to-noise ratio. This effect can be significantly pronounced in ViTs while LLMs can lack visible noise. Therefore, the ^^-LRP rule is only applied to linear layers in ViTs. In the following, the handling of element-wise non-linearities will be explained. Since element- wise non-linearities have only a single input and output variable, the decomposition of equation (1) is the operation itself. Therefore, the entire incoming relevance ^^^^can only be assigned to the single input variable. ^^^^-^ൌ^^^^(9) The identity rule (9) can be applied to all element-wise operations with a single input (e.g., predecessor node 107) and single output variable (e.g., predetermined node 106). In the following, the attributing of Non-linear Attention according to embodiments of the invention is explained. The heart of the transformer architecture (Vaswani et al., 2017) is non- linear attention where ^^^ denotes matrix multiplication (e.g., a dot product 114), ^^ ∈ ℝ^ൈ^ൈௗೖ is the keymatrix, ^^ ∈ ℝ^ൈ^ൈௗೖ is the queries matrix, and ^^ ∈ ℝ^ൈ^ൈௗೡ the values matrix, and ^^ ∈ℝ^ൈ^ൈௗೡis the final output of the attention mechanism. ^^ is the batch dimension including the number of heads, and ^^^, ^^௩indicate the embedding dimensions, and ^^^, ^^^are the number of query and key / value tokens. First and foremost, the softmax function (e.g., softmax function 115) is highly nonlinear. In addition, the matrix multiplication (e.g., dot products 114) is bilinear, i.e., linear in both of its input variables (e.g., attention matrix 111 and value matrix 112). In the following, relevance propagation rules for each of these operations according to embodiments of this invention are derived, taking into account considerations of efficiency. In the following, the handling of softmax non-linearity (e.g., of a softmax function 115) according to embodiments of this invention is explained. Above, a generalized approach to linearization that incorporates bias terms has been explained, allowing for the absorption of a portion of the relevance (e.g., relevance score 103). However, (Ali et al., 2022) advocates for a strict adherence to the conservation property (3) and argues that a linear decomposition of FH240201PEP-2025036018.DOCX a non-linear function should typically exclude a bias term. While the virtue of this approach for operations such as RMSNorm (Zhang and Sennrich, 2019) or matrix multiplication, where ^^^0^ ൌ 0 can be seen, it can be argued, that a linearization of the softmax function (e.g., softmax function 115) should inherently incorporate a bias term. This is due to the fact that even when the input (e.g., a single node of the attention matrix 11 or the value matrix 112) is ^ zero, the softmax function (e.g., softmax function 115) yields a value of ே (where ^^ represents the dimension of the inputs) which is analogous to a bias term. According to embodiments of the invention, decomposing the softmax function (e.g., softmax function 115) by a Taylor decomposition (4) at reference point x may yield the following relevance propagation rule (e.g., for relevance scores 103): ^^^^ where ^^^denotes the ^^-th output of the softmax function (e.g., softmax function 115). The hidden bias term, which represents the approximation error, consequently absorbs a portion of the relevance (e.g., relevance score 103). The proof can be found later. Note, that the works (Voita et al., 2021; Chefer et al., 2021b; Ali et al., 2022) propose to handle the bias term differently to strictly enforce the conservation property (3). (Voita et al., 2021) linearizes at ^^ but distributes the bias term equally on all input variables, while (Chefer et al., 2021b) applies the element-wise identity rule (9) and hence omits the bias term completely. Both variants can lead to severe numerical instabilities as discussed later. Finally, (Ali et al., 2022) regards the attention matrix ^^ (e.g., attention matrix 111) in equation (11) as constant, attributing relevance solely through the value path and disregarding the softmax operation entirely. The following explains the Handling of Matrix-Multiplications (e.g., dot product 114) according to embodiments of the invention. Since ^^^0, 0^ ൌ 0, it is desirable to decompose the matrix multiplication (e.g., dot product 114) without a bias term. To achieve this, we break down the matrix multiplication (e.g., dot product 114) into an affine operation involving summation and a bi-linear part involving element-wise ୠ୧ି୪୧୬^ୟ୰ ୮ୟ୰^ The summation provides a decomposition in the form of equation (1), and the individual summands ^^^^^^^^(e.g., individual nodes of the attention matrix 111 and the value matrix 112) can be decomposed. Since multiplication is a commutative operation, assigning equal relevance values to both operands is reasonable. FH240201PEP-2025036018.DOCX According to embodiments of the invention, decomposing element-wise multiplication with N input variables of the form ^^^^^^^ൌ∏ே ^ ^^^by Shapley (with baseline zero) or Taylor decomposition (4) at reference point x (without bias or distributing the bias uniformly) yields the following uniform relevance propagation rule: ^ ^^^. The proof can be found later. using: Decomposing matrix multiplication with a sequential application of the uniform rule (14) and the ^^-rule (8) can, according to embodiments, yield the following relevance propagation rule: ^ There is no bias term For ^^^^, one can sum over the ^^ indices. The proof can be found later. By employing this rule, strict adherence to the conservation property (3) can be maintained, while also preserving numerical stability. The following explains the handling of normalization layers (e.g., normalization function 117) according to embodiments of the invention. Commonly used normalization layers (e.g., normalization function 117) in Transformers include LayerNorm (Ba et al., 2016) and RMSNorm (Zhang and Sennrich, 2019). These layers apply affine transformations and non- linear normalization sequentially. (16)(17) where ^^, ^^^ , ^^^ ∈ ℝ. Affine transformations such as the multiplicative weighting of the output orthe subtraction of the mean value are linear operations that can be attributed by the ^^-LRP rule. Normalization (e.g., normalization function 117), on the other hand, is non-linear and ே^The work (Ali et al., 2022) demonstrates that when linearizing LayerNorm at ^^, the bias term can absorb most of the relevance equal to ^^^^^^^^^^ / ^^^^^^^^^^^ ^ ^^^, effectively absorbing 99% of the relevance with commonly used values of ^^ ൌ 10-^and ^^^^^^^^^^ ൌ 1. Hence, a linearization at ^^ is not meaningful. As a solution, (Ali et al., 2022) proposes to skip this operation with the FH240201PEP-2025036018.DOCX identity rule. In the following, it is shown, that according to embodiments of the invention, this heuristic can be derived from the Deep Taylor Decomposition framework. According to embodiments of the invention, decomposing LayerNorm or RMSNorm by a Taylor decomposition (4) with reference point 0 (without bias or distributing the bias uniformly) can yield the identity relevance propagation rule: ^^^ି^^ ^^^^There is no bias that absorbs relevance. This rule enforces a strict notion of conservation, while being highly efficient by excluding normalization operations (e.g., normalization function 117) from the computational graph. Experiments disclosed herein provide evidence that this simplification is faithful. In the following, the understanding of latent features according to embodiments of the invention is explained. For Convolutional Neural Networks (CNNs) (e.g., a ML predictor 101), single units (i.e., neurons or filters) have been shown to correspond to distinct concepts, each fulfilling specific sub-tasks (Radford et al., 2017; Achtibat et al., 2023). In order to find the most representative reference samples that explain the neuron’s encoding, researchers rely on ActMax (Nguyen et al., 2016), where input samples are sought that give rise to the highest activation value. It is followed up on these observations and the following strategy for understanding latent features according to embodiments of the invention is presented: (1) Collect prompts (e.g., prompts 108) that lead to the highest activation (e.g., activation 104) of a unit. (2) Explain the unit’s activation (e.g., activation 104) using AttnLRP, allowing to narrow down the relevant input tokens for the chosen unit. In this case, a unit or a knowledge neuron can, for example, be any of the following: a predecessor node 107, a predetermined node 106, an element / node of an operand 111, an element / node of a further operand 112, an element / node of a matrix 113, an element / node of a matrix 113’, a node 118, and also multiple of these considered together. In the following the description concentrates on knowledge neurons (Dai et al., 2022; Voita et al., 2023) that are situated at the last non-linearity in FFN layers ^^^ൌ ^^^^^^^^^^^^^^^. These neurons possess intriguing properties, as shown in Fig. 3b: They encode factual knowledge and upon activation, the corresponding row of the second weight matrix ^^ଶis added to the residual stream directly influencing the output distribution of the model. By projecting this weight row onto the vocabulary, a distribution of the most probable tokens across the vocabulary is obtained (Geva et al., 2022). Applying AttnLRP on ActMax reference samples FH240201PEP-2025036018.DOCX and projecting the weight row on the vocabulary allows an understanding in which context a neuron activates and how its activation influences the prediction of the next token. In contrast to (Ali et al., 2022), AttnLRP also allows analyzing the key and value linear layers (e.g., value matrix 112) inside attention modules. The description will proceed with an explanation of experiments that have been done to evaluate embodiments of the proposed inventive concept. The experiments aim to answer the following questions: •How faithful are explanations using AttnLRP compared to other state-of-the-art approaches? (Q1) •How efficient is AttnLRP compared to perturbation-based methods? (Q2) •Can we understand latent representations and interact with LLMs (e.g., ML predictor 101)? (Q3) The description now proceeds with an evaluation of explanations, related to (Q1). A reliable measure of faithfulness of an explanation are input perturbation experiments (Samek et al., 2017). This approach iteratively substitutes the most important tokens in the input domain with a baseline value. If the attribution method accurately identified the most important tokens, the model’s confidence (e.g., the ML predictors 101 confidence) in the predicted output should rapidly decrease. The other way around, perturbing the least relevant tokens first, should not affect the model’s prediction (e.g., the ML predictors 101 output 110) and result in a slow decline of the model’s confidence. Despite its drawbacks, such as potentially introducing out- of-distribution manipulations (Chang et al., 2018) and sensitivity towards the chosen baseline value, this approach is widely adopted in the community. (Blücher et al., 2024) have addressed this criticism and introduced an enhanced metric by quantifying the area between the least and most relevant order perturbation curves to obtain a robust measure. Hence, this improved metric has been employed to measure faithfulness. In order to assess plausibility, the SQuAD v2 Question-Answering (QA) dataset (Rajpurkar et al., 2018) is utilized, which includes a ground truth mask indicating the correct answer within the question. Attributions for accurately answered questions are calculated and the Intersection over Union (IoU) between the positive attribution values and the ground truth mask can be determined. This approach assumes that the model (e.g., ML predictor 101) solely relies on the information provided in the ground truth mask, which is not entirely accurate but sufficient for identifying a trend. FH240201PEP-2025036018.DOCX The description now proceeds of an evaluation of the baselines. The faithfulness on two self- attention models (e.g., ML predictor 101) are evaluated, a ViT-B-16 (Dosovitskiy et al., 2021) (e.g., ML predictor 101) on ImageNet (Deng et al., 2009) classification and the Llama 2-7b (Touvron et al., 2023) model (e.g., ML predictor 101) on IMDB movie review (Maas et al., 2011) classification as well as next word prediction of Wikipedia (Wikimedia Foundation, 2023). To evaluate cross-attention layers, the encoder-decoder model Flan T5-XL (Chung et al., 2022) on the SQuAD v2 dataset is used. The proposed inventive concept is denoted as AttnLRP and is compared against a broad spectrum of methods including InputൈGradient (IൈG), Integrated Gradients (IG), SmoothGrad (SmoothG), Attention Rollout (AttnRoll), Gradient-weighted Attention Rollout (GൈAttnRoll) and Conservative Propagation (CP)-LRP. Embodiments of the present inventive concept propose to apply the ^^-rule in the case of ViTs. For better comparison, an enhanced CP-LRP baseline is also included, which also uses the ^^-rule in the ViT experiment. The LRP variants introduced by (Voita et al., 2021; Chefer et al., 2021b) are excluded due to numerical instabilities observed in preliminary experiments. Further, the Grad- CAM adaptation is utilized, as described in (Chefer et al., 2021b). Specifically, the last attention map is weighted with the gradient. Finally, an expansion upon AtMan is done by incorporating it into encoder-decoder models by suppressing tokens in all self-attention layers within the encoder, while only doing so in cross-attention layers within the decoder. For AtMan, SmoothGrad and Rollout-methods a hyperparameter sweep over a subset of the dataset is performed to find optimal suppression and noise factors. Fig. 4a depicts a table 480, that compares the IoU of different methods 484 on different transformer ML predictors (e.g., ML predictor 101), namely ViT-B-16481 on the ImageNet dataset, Llama 2-7b 482 on the IMDB and Wikipedia Dataset and Flan-T5-XL on the SQuaDv2 483 and SQuADv2 IoU dataset. The table 480 depicts faithfulness scores as area between the least and most relevant order perturbation curves on different models 481, 482, 483 (e.g., ML predictor 101) and datasets. To assess plausibility, the IoU in the SQuAD v2 dataset is depicted. The higher a score, the better. AttnLRP consistently outperforms all the state-of-the- art methods in terms of faithfulness. It can be also observed, that gradient-based approaches significantly suffer from noisy attributions, as reflected by the low faithfulness scores. CP-LRP with ^^ applied on all layers, also suffers from noisy gradients. Applying the ^^-rule for CP-LRP and AttnLRP in ViTs improves the faithfulness substantially. Whereas AtMan does not perform well in unstructured tasks, i.e., next word prediction or classification, it achieves a high score in QA tasks. While GൈAttnRoll better reflects the model behavior in unstructured tasks compared to AtMan, it is affected by considerable background noise, resulting in a low IoU score in the SQuAD v2 dataset FH240201PEP-2025036018.DOCX The description proceeds with a discussion of the computational complexity 487 and memory consumption 489, relating to (Q2). Fig. 5 depicts a table 580, the table 580 shows the computational complexity 487 and memory consumption 489 for different methods 484, namely of LRP-based methods 485, such as AttnLRP, and linear-time perturbation methods 486 measured w.r.t. a single forward pass. ^^^: number of layers, ^^்: number of tokens. The Table 580 illustrates the computational complexity 487 and memory consumption 489 of a single LRP-based attribution (LRP based methods 485) and linear-time perturbation methods 486, such as AtMan or a Shapley-based method (Fatima et al., 2008). Linear-time perturbation methods 486 require ^^்forward passes, but have only a memory consumption 489 of ^^^1^. Since LRP (e.g., AttnLRP) is a backpropagation-based method, gradient checkpointing (Chen et al., 2016) techniques can be applied. In checkpointing, LRP -based methods 485 (e.g., AttnLRP) require two forward and one backward pass, while the memory consumption 489 scales sublinear with the number of layers. The description proceeds with a discussion of understanding and manipulating neurons, relating to (Q3). In the investigation, the Phi-1.5 model (Li et al., 2023) has been used (e.g., as a ML predictor 101), which has a transformer-based architecture with a next-word prediction objective. Reference samples for each knowledge neuron have been obtained by collecting the most activating sentences over the Wikipedia summary dataset (Scheepers, 2017) (e.g., the node with the highest activation 104). Fig.3a. shows how combining AttnLRP with ActMax allows to identify relevant neurons and gain insights into their encodings. This allows one to manipulate the latent representations and (e.g., using an apparatus 1440 or 1550), e.g., to change an output 110 “Arctic” (by disabling a corresponding neuron 3791) to “Desert” by activating a neuron 3792or “Candy Store” by activating a neuron 3793In Fig.3a-b a neuron 379 can, for example, be any of the following: a predecessor node 107, a predetermined node 106, an element / node of the operand 111, an element / node of the further operand 112, an element / node of a matrix 113, an element / node of a matrix 113’, a node 118, and also multiple of these considered together. To illustrate, the using of a prompt 108: ‘The ice bear lives in the’ at the input 109 gives the corresponding prediction at the output 110: ‘Arctic’. Using AttnLRP, the most relevant layers for predicting ‘Arctic’ as well as the specific neurons within the FFN 378 (e.g., the FFN can be a ML predictor 101 or a portion 102) layers contributing to this prediction can be determined. The analysis reveals that the most relevant neurons 3791,3792and 3793after the first three layers are predominantly situated within the middle layers. Notably, one standout neuron 3791(e.g., the #3948 in layer 17) FH240201PEP-2025036018.DOCX activates on reference samples about cold temperatures, as depicted in Fig. 3b. This observation is further validated by projecting the weight matrix of the second FFN 378 layer onto the vocabulary. The neuron shifts the output distribution of the model to cold places, winter sports and animals living in cold regions. Analogously, for the prompt ‘Children love to eat sugar and’ with the prediction ‘sweets’, a most relevant neuron’s 3793(e.g., layer 18, neuron #5687) projection onto the vocabulary signifies a shift in the model’s focus towards the concept of candy, temptation and sweetness in the vocabulary space. One can interact with the model by deactivating 3751neuron 3791, and strongly amplifying the activation 3763of neuron 3793in the forward pass. This manipulation yields the following prediction change: Prompt 108: Ice bears live in the Prediction (e.g., Output 1103): sweet, sugary treats of the candy store. It can be further noticed, that neuron 3792(e.g., #4104 in layer 17) encodes for dryness, thirst and sand. Amplifying its activation 3762changes the output 1102to ‘desert’, as illustrated in Fig.3a. AttnLRP, enables tracing the most important neurons 379 in models (e.g., ML predictor 101) with billions of parameters. This allows to systematically navigate the latent space to enable targeted modifications to reduce the impact of certain concepts (for example, ‘coldness’) and enhance the presence of other concepts (for example, ‘dryness’), resulting in discernible output changes (e.g., at an output 110). Such an approach holds significant implications for transformer-based models (e.g., ML predictor 101), which have been difficult to manipulate and explain due to inherent opacity and size. Fig.3b shows two approaches for understanding knowledge neurons: (a) a Neuron 379 in a FFN 378 (e.g., the 3948-th Neuron at the last non-linearity in FFN layer 17) of the Phi-1.5 model selects a weight row 377 to add to the residual stream. This weight row 377 projected on the vocabulary spans topics about ice, cold places and winter sport. (b) Sentences 376 that maximally activate this neuron 379 contain references about coldness. Attributing the neuron 379 with AttnLRP can highlight the most relevant tokens inside the input sentences 376. Note, although not illustrated here, AttnLRP can also attribute linear layers inside attention modules. The description proceeds with deriving conclusions about embodiments of the present inventive concept. A Layer-wise Relevance Propagation framework is presented, that is accustomed to non-linear attention, novel rules for the softmax and matrix-multiplication step are proposed and interpretations in terms of Deep Taylor Decomposition are provided. The AttnLRP method stands out due to its unique combination of simplicity, faithfulness, and efficiency. Its applicability both for LLMs as well as ViTs are demonstrated, utilizing the FH240201PEP-2025036018.DOCX stabilizing effect of the ^^-rule. In contrast to other backpropagation-based approaches, AttnLRP enables the accurate attribution of neurons in latent space (also within the attention module), thereby introducing novel possibilities for real-time model interaction and interpretation.   The description proceeds with an impact discussion of embodiments of the present inventive concept. The presented embodiments establishes the foundations that make it possible to systematically analyze and debug transformer-based AI systems (e.g., ML predictor 101), thereby minimizing the occurrence of false or misleading outputs (hallucination) (e.g., at an output 110) and mitigating biases that may arise from training data or algorithmic processes. Particularly, it opens up the door for future applications of transformer-based AI systems in critical domains such as healthcare and finance, where the ability to explain the model behavior is often a (legal) requirement. The high computational efficiency of AttnLRP significantly reduces the energy usage and consequently also the financial overhead associated with the explanation, which will result in a broader adoption of XAI for transformers. The following provides further details on the methods and apparatuses according to embodiments presented herein. Implementation details are provided and the stability of the bias term and the noise problem in Vision Transformers are discussed. Also, proofs are provided for the four propositions presented above. In the following, details of embodiments according to the present inventive concept are presented. This section provides more details on AttnLRP and justifies the specific parameter choices made in this work (e.g., use of ^^-LRP in Vision Transformers). In the following, Conservation & Numerical Stability of Bias Terms is discussed. The relevance (e.g., relevance score 103) of a specific neuron or function, denoted as ^^^, can be computed by summing the contributions of the input variables, represented by ^^^^^^^, and adding thecontribution of the bias term, ^^^^. ^ೕ^௫^The relevance of the bias term itself, be calculated as: ^^ೕ^௫^If we want to compute the relevance of the input variables ^^^while ensuring strict adherence to the conservation property (3), we can exclude the bias term, so that it does not absorb part of the relevance ^^^. There are at least three options: One can completely omit the bias term, distribute its relevance value uniformly across the input variables, or apply the identity rule. FH240201PEP-2025036018.DOCX Omitting the bias term: In this case, the relevance propagation equation can be:^^^ ൌ ∑^ ^^^←^ ൌ ∑^ ^^^^ ^^ோೕ^one no longer divides by the original function ^^^^^^^. no sign flips occur, as∑^^^^^^^^might have a different sign than ^^^^^^^. Distributing the bias term: Alternatively, the relevance value of the bias term can be distributed uniformly across the input variables. The relevance propagation equation in this case is: ^^ ^^ ^^^ Here, ^^ represents the number this approach may lead to numerical instabilities if the function ^^^^^^^ൌ0 with ^^^ൌ0. Then, the relevance message ^^^←^is: ^^ Since ^^ is very small, this term can explode and cause numerical instabilities. Thus, functions that fulfill ^^^0^ൌ0 should preferably be linearized such that the bias term does not cause numerical instabilities. Applying the identity rule: The identity rule can be applied as follows: ^^^ି^^ ൌ ^^^^However, one may encounter numerical instabilities, but these effects will only become visible in the next sequential propagation rule, not at this layer yet. For example, in the softmax function, one may encounter a situation where a ^^^^-^ൌ0 but ^^^^^^^^^0. If a non-zero relevance value from layer ^^ is assigned to ^^^^^^^^, then its relevance value is propagated through the identity rule to ^^^^-^. Assuming the ^^-rule is applied after the identity rule, the relevance is given by: ^^ିଶ,^ି^^ ^ ோ^షభ ^^←^ ൌ ^ ିଶ ^^ ^^^ ^^^^ାఌ Note, the division by ^^^^-^ൌ^^^^-^ൌ0, which results in numerical instabilities. Further note, that these instabilities would not occur if ^^^^-ଶൌ0 or ^^^^-^ൌ0, which is the case for a function fulfilling ^^^^^0^ൌ0. In summary, omitting the bias term completely, distributing its relevance value uniformly across the input variables or applying the identity rule are possible approaches, but they have their considerations and potential challenges. Regarding the softmax non-linearity, (Voita et al., 2021) distributes the bias term equally on all input variables, while (Chefer et al., 2021b) apply FH240201PEP-2025036018.DOCX the element-wise identity rule (9). Both variants can lead in this case to severe numerical instabilities. In the following, the difference between various LRP methods is discussed. In Fig.13 a table 1380 is shown, comparing different methods 484, the methods 484 are related to relevance score assignment, and their approach to different operations of a ML predictor, namely a softmax function 115, a dot product 114 (exemplarily illustrated as matrix multiplication) and LayerNorm as an example for a normalization function 117. In the Table 1380 the different strategies are illustrated, that have been employed for LRP in the past. As discussed previously, distributing the bias term uniformly or applying the identity rule on the Softmax function 115 can lead to numerical instabilities. (Chefer et al., 2021b) applies a normalization by dividing both arguments by the summation of its absolute values. (Ali et al., 2022) regards the Softmax 115 as constant and does not propagate relevance through it, thereby circumventing the issue completely. Applying the ε-rule on matrix multiplication (e.g., the dot product 114) does violate the conservation property (3) as proved in Lemma 3 of (Chefer et al., 2021b). In the following, Tackling Noise in Vision Transformers is discussed. Since backpropagation- based attributions utilize the gradient, they may produce noisy attributions in models with many layers, where gradient shattering and noisy gradients appear (Balduzzi et al., 2017; Dombrowski et al., 2022). Hence, various adaptions of the ^^-LRP rule were developed to strengthen the signal-to-noise ratio by dampening counter-acting activations (Bach et al., 2015; Montavon et al., 2019). Here, the generalized ^^-rule that encompasses all other proposed rules in the literature (Montavon et al., 2019) can be used. Let ^^^^be the contribution of input ^^ to output ^^, e.g., ^^^^^^^, and ^^^the neuron output activation (e.g., of the node 118^^ as illustrated in . Then on the of ^^^: ^^ ^^ ∞,LRP-^^ା-rule, ^^ ^←^௭శೕ by only taking into account positive contributions ^^^^^^^^^^^^^with ^⋅^^ൌ^^^^^^^0,⋅^. This is the case of softmax layers (e.g., of the attention layer) where only positive values are dealt with. FH240201PEP-2025036018.DOCX Remarkably, observations of the inventors reveal that attributions in LLMs (e.g., ML predictor 101) demonstrate high sparsity and lack visible noise, while ViT (e.g., ML predictor 101) are susceptible to gradient shattering. The inventors hypothesize that the discrete nature of the text domain may affect robustness (Mao et al., 2021). Therefore, the ^^-rule is only applied in ViT (e.g., ML predictor 101) in the convolutional and linear layers. To further increase the faithfulness, the ^^-rule can be also applied on softmax layers (e.g., softmax function 115). Since the output of the softmax (e.g., softmax function 115) is always greater than zero, the simplified ^^^-rule can be applied (special case of ^^-rule). In the following, proofs for the rules presented herein are provided. Following, the decomposition of the softmax function (e.g., softmax function 115) by linearizing (4) it at ^^ is demonstrated. This can be started by considering the softmax function (e.g., softmax function 115): The derivative of the softmax (e.g., softmax function 115) has two cases, which depend on the output and input indices ^^ and ^^: ^^Consequently, a Taylor decomposition (4) yields: Two cases are differentiated, namely (i) when relevance from output ^^ to input ^^ ് ^^ is attributedand (ii) when attributing from output ^^ to input ^^ൌ^^. ^^^ ^^Applying equation (5), one can obtain: ^ In the following, decomposing multiplication (e.g., dot product 114) is discussed, starting with the decomposing of the multiplication of ^^ ^ ^^^It starts by performing a Taylor Decomposition (4), then the same decomposition with Shapley is derived. Taylor Decomposition: The derivative is డ௫^ൌ∏^ஷ^^^^FH240201PEP-2025036018.DOCX Consequently, a Taylor decomposition (4) at ^^ yields ^^^^^^^ ൌ ∑ேడ^ೕ^^^^ ^ ^^^^ ൌ ^^∏ே ^^^^ ^ ^^^^ ൌ ^^^^^^^^^ ^ ^^^^One can either omit the to strictly enforce the conservation property (3). Here, it is demonstrated how to distribute the bias term uniformly. ^^^^^^^1For omitting the bias term, Shapley: The Shapley value (Lundberg and Lee, 2017) is defined as: ^^ ^^^^|ௌ|!^ேି|ௌ|ି^^!^^^^^^ ^^^^^ ^^^^^^^ where ^^^^^^^ is the Shapley all features, and ^^ denotes a feature subset (coalition). With respect to multiplication, zero is the absorbing element. Hence, zero is chosen as baseline value, and the Shapley value function becomes: ^^^^^ ∪ ^^^^^ൌ∏^ ^^^ The symmetry theorem (Fryer et al., 2021) of Shapley states that the contributions of two feature values ^^ and ^^ should be the same if they contribute equally to all possible coalitions ^^^^^ ∪ ^^^^^ൌ^^^^^ ∪ ^^^^^∀^^ ⊆ ^1,2, ... ^^^^^^, ^^^then ^^^^^^^ൌ^^^^^^^. In addition, the efficiency theorem (Fryer et al., 2021) states that the output contribution is distributed equally amongst all features. Hence, the output contribution is equal to the sum of coalition values of all features ^^, ∑^^^^^^^^ൌ^^^^^^ Both theorems are applicable and hence it follows: ே In the case of LRP (e.g., AttnLRP), ^^^^^^ is identified as ^^^. The description proceeds with discussing the decomposing of bi-linear matrix multiplication (e.g., dot product 114). Consider the equation for matrix multiplication (e.g., dot product 114), where the terms are treated as single input variables: ^^^^ൌ∑^൫^^^^^^^^൯ In this case, the function already is in decomposition (1). Therefore, ைೕ^FH240201PEP-2025036018.DOCX Next, the individual terms can be decomposed using the derived rule from above: ^^ ൌ∑ ∑ ^ ∑ோೕ^^^ ^^^^^←^^^ൌ^^^^^^ൌ^^^^^^^^^The proof for ^^^^follows a In the following, layer normalization (e.g., normalization function 117) is discussed. Consider layer normalization (e.g., normalization function 117) of the form ^^^^^௫ೕ^^ൌ ^^௫^ In LayerNorm (Ba et al., 2016) (e.g., normalization function 117), for simplicity ^^^^^^ ൌ 0 isassumed, then the partial derivative simplifies to Further, the partial derivative of RMSNorm (Zhang and Sennrich, 2019) (e.g., normalization function 117) is At reference point ^^^ 0, the diagonal elements in equation (27) ^^ ് ^^ are zero, yieldingTaylor decomposition: To enforce a strict notion of the conservation property (3), the bias term~^^^can be excluded or evenly distributed across the input variables (e.g., multiple nodes 118^^ି^). Because we have only a single input variable (e.g., activation of the node 118^^ି^, the bias can be considered as part of ^^^. ^ Since there is only one input variable (e.g., activation of the node 118^^ି^) and one output (e.g., activation of the node 118^^), the decomposition is equivalent to the identity function, as discussed above about component-wise non-linearities. Thus, we conclude that the identity rule applies in this case. ^^^ି^^ ൌ ^^^^(29)Note, that this rule is numerically stable because ^^^^0^ ൌ 0 as discussed previously.FH240201PEP-2025036018.DOCX In the following, additional details about the experiments performed are provided. In the following, models and datasets are discussed. For ImageNet faithfulness, the pretrained Vision Transformer 16-B weights of the PyTorch model zoo (Paszke et al., 2019) (e.g., ML predictor 101) are utilized. 3200 samples were randomly selected such that the standard error of mean converges to below 1% of the mean value. For Wikipedia, SQuAD v2 and IMDB faithfulness, the pretrained Llama 2-7b (e.g., ML predictor 101) and Flan-T5 (e.g., ML predictor 101) weights hosted on huggingface (Wolf et al., 2019) were evaluated on 4000 randomly selected validation dataset samples (fixed set for all baselines). Both models (e.g., ML predictor 101) are casted to the Brain Floating Point (BFLOAT16) half-precision format to save memory consumption and to simulate real-world scenarios. Further, for Wikipedia next word prediction the model (e.g., ML predictor 101) was evaluated on a context size of 512 (from beginning of article until context length is reached), while the context size in SQuAD v2 varies between 169 to 4060. Although Flan-T5 was trained on a smaller context size of 2000 tokens, the relative positional encoding allows it to handle longer context sizes with at least 8192 tokens (Shaham et al., 2023). Additionally for SQuAD v2 faithfulness and IoU, the following prompt was utilized: Context: [text of dataset sample] Question: [question of dataset sample] Answer: In order to prevent out-of-distribution samples, only the tokens inside the context [text of dataset sample] were perturbed. Since the length of the SQuAD v2 questions varies and the length of the validation dataset is small, the standard error of mean is higher in table 780a. For IMDB, a last linear layer was added to Llama 2-7b (e.g., ML predictor 101) and the model (e.g., ML predictor 101) was finetuned, which achieves 93% accuracy on the validation dataset. If NaN values for a sample were encountered, it was removed from the evaluation. This happened for Grad ൈ AttnRollout and AtMan in the Wikipedia dataset. However, the standard error of the mean remains small, as can be seen in Table 4. The following parameters were used during evaluation, relating to benchmarking energy, time and memory consumption. Benchmarking on a Node with four A10040GB and CPU? and 16 threads, 512 GB CPU RAM and AMD EPYC 73F33.5 GHz because attention rollout, atman and LRP (e.g., AttnLRP) needs access to the attention weights, we didn’t use flash attention for fair comparison The description proceeds with an outline of AttnLRP Composites for ViT. Applying the ^^-rule on all linear layers inside LLMs (e.g., ML predictor 101) can be sufficient to obtain faithful and FH240201PEP-2025036018.DOCX noise-free attributions. However, for the Vision Transformers (e.g., ML predictor 101), the ^^- rule is applied on all linear layers (including the convolutional layers) outside the attention module. Since the ^^-rule has a hyperparameter, the work (Pahde et al., 2023) proposed to tune the parameter using a grid-search. This optimization search (or in LRP known as composite search) is computational highly demanding. The Vision Transformer (e.g., ML predictor 101) consists of many linear layers. The proposed approach is to use different ^^ values across different layer types. According to (Vaswani et al., 2017) the attention module consists of several linear layers which are referred to as LinearInputProjection: ^^ൌ^^^^^^^^^^^ൌ^^^^^^^^^^^ൌ^^௩^^^^^௩In the attention layer, after the utilization of softmax (e.g., softmax function 115), based on the equation (11), there exists another linear layer with the below format noted by LinearOutputProjection: ^^ൌ^^^^^^^^^The other layers in the whole network (e.g., ML predictor 101), will be referred to as Linear. The perturbation experiment had been conducted over these layers using different types of rules including Epsilon, ZPlus, Gamma, and AlphaBeta (with ^^ൌ2 and ^^ൌ1 according to (Montavon et al., 2019)). The best found composite which has been used for AttnLRP and CP- LRP in tables 480, has the setting as proposed in table 580. More details over the statistics of the conducted experiments are available in the Fig.9a-e. Fig. 2 shows a broad comparison of the capabilities of an apparatus according to an embodiment of the invention with SmoothGrad, GradxAttnRoll and AtMan. Fig. 2 may be summarized with the following: By optimizing LRP for transformer-based architectures, AttnLRP outperforms other state-of-the-art methods in terms of explanation faithfulness and computational efficiency. AttnLRP further is able to explain latent neurons inside and outside the attention module, allowing interactions with the model. Legend: highly (+), semi- (◦), not suited (−). Credit: Nataba / iStock. Fig.6 shows a comparison of the AttnLRP 669 (e.g., Attn-LRP) with GradAttnRoll 668 and can be described with the following: Comparison of the ^^Attn-LRP (ours) and GradൈAttnRoll (Chefer et al., 2021a) techniques through the perturbation experiment (Faithfulness) on the ViT using test set of ImageNet. From left to right, the plots correspond to ^^^^^^^ி^ோி^-^^^^^^ெி^ோி^ FH240201PEP-2025036018.DOCX 667, ^^^^^^ெி^ோி^ 666, and ^^^^^^^ி^ோி^ 665. The term “AUC”, is also equivalent to (left to right order) ^^^^^^, ^^ி^^ோி, and ^^ிெ^ோி. Fig.7a-d show evaluations of faithfulness for different methods 484 on different datasets. Fig. 7a shows a table 780a, that can be explained the following. ViT Perturbation Experiment (Faithfulness), SmoothGrad’s ^^ ൌ 0.01, AtMan (1.0, 0.1). ”all epsilon” indicates that the ^^-rule has been used to propagate relevance to the layers. The term ”best” refers to the utilization of LRP via the composite proposed in 8.1.5. ^^^^ி790 demonstrates the area under the curve for the perturbation experiment of the type Flipping which leverages insights from both ^^ிெ^ோி791 of the most relevant first order, and ^^ி^^ோி792 of least relevant first order. (^^^^ிൌ ^^ி^^ோி- ^^ிெ^ோி) The higher the ^^^^ி790, the better. Relevance scores (e.g., relevance scores 103) obtained using an apparatus according to an embodiment are performing best. Fig.7b shows a table 780bthat shows an evaluation of faithfulness for different methods 484 on the Wikipedia dataset. Wikipedia Perturbation Experiment (Faithfulness), SmoothGrad ’s ^^ ൌ 0.05, AtMan. ”all epsilon” indicates that the ^^-rule has been used to propagate relevance to the layers. ^^^^ூ793 demonstrates the area under the curve for the perturbation experiment of the type Insetion (e.g., in other words insertion) which leverages insights from both ^^ூெ^ோி794 of the most relevant first order, and ^^ூ^^ோி795 of least relevant first order. (^^^^ூൌ ^^ூெ^ோி- ^^ூ^^ோி) The higher the ^^^^ூ793, the better. Relevance scores obtained using an apparatus according to an embodiment are performing best. Fig.7c shows a table 780cthat shows an evaluation of faithfulness for different methods 484 on the IMDB dataset. IMDB Perturbation Experiment (Faithfulness), SmoothGrad’s σ = 0.05, AtMan 0.7. ”all epsilon” indicates that the ε-rule has been used to propagate relevance to the layers. ^^^^ூ793 demonstrates the area under the curve for the perturbation experiment of the type Insertion which leverages insights from both ^^ூெ^ோி794 of the most relevant first order, and ^^ூ^^ோி795 of least relevant first order. (^^^^ூൌ ^^ூெ^ோி- ^^ூ^^ோி) Fig.7d shows a table 780dthat shows an evaluation of faithfulness for different methods 484 on the Squad dataset. Squad Perturbation Experiment (Faithfulness), SmoothGrad with optimal sigma, AtMan 0.9. ”all epsilon” indicates that the ^^-rule has been used to propagate relevance to the layers. ^^^^ூ793 demonstrates the area under the curve for the perturbation experiment of the type Insetion (e.g., in other words, insertion) which leverages insights from both ^^ூெ^ோி794 of the most relevant first order, and ^^ூ^^ோி795 of least relevant first order. (^^^^ூൌ ^^ூெ^ோி-^^ூ^^ோி) FH240201PEP-2025036018.DOCX Fig. 8 shows a table 880, that shows an overview of proposed composites according to embodiments of the invention. The table 880 shows for different layer types 896 the proposed rules 897.In other words, the table 880 shows Proposed Composite for the AttnLRP and CP- LRP methods. Fig. 9a-e show statistics for different layer type, comparing different relevance propagation rules Fig.9a shows a statistics on rules used for softmax layers: Propagating relevance values through (specifically by applying ZPlus(e.g., ^^ାrule) rule) Softmax marginally improves the faithfulness of explanation compared to the cases where we block its propagation. Fig. 9b shows statistics on rules used for convolution layers: Applying ZPlus (e.g., in other words, ^^ାrule) and AlphaBeta proposes acceptable results however the most faithful results can be reached via Gamma (^^= 0.25). Fig.9c shows statistics on rules used for linear layers: Similar to Convolution layers, Gamma seams more promising however with different ^^ value (0.05 in this case). Fig. 9d shows statistics on rules used for LinarInputProjection (in other words, LinearInputProjection) layers: Gamma and Epsilon rules are competitive in this case, however since there is larger difference between the minimum and the lower quartile in Gamma rules, the most faithful choice will be Epsilon. Fig. 9e shows statistics on rules used for LinarOutputProjection (in other words, LinearInputProjection) layers: The Epsilon rule outperforms other rules clearly. The description proceeds with a discussion of attributions of knowledge neurons. Fig. 10a-c illustrate the top 10 sentences in the Wikipedia summary dataset that maximally activate a knowledge neuron. We applied AttnLRP to highlight the tokens inside these reference samples. We observe that knowledge neurons exhibit remarkable disentanglement, e.g., neuron #256 of layer 18 shown in Figure 4 seems to encode concepts related to transport systems (railways in particular), while neuron #2207 of layer 20 shown in Figure 5 seems to encode the concept teacher, in particular a teacher, in an unusual context (e.g., inappropriate behavior, sexual misconduct). The degree of disentanglement should be studied in future work. FH240201PEP-2025036018.DOCX Fig. 10a shows a set of prompts 1099agiven to a model, where using AttnLRP the corresponding relevance scores of a specific neuron hast been marked in the prompts using different highlighting effects. It can be seen, that the respective neuron is related to transport systems, especially railways. In other words, Fig.10a shows AttnLRP Attribution for a neuron (e.g., #256, in layer 18). The knowledge neuron seems to activate for transport systems (railways in particular). Fig. 10b shows a set of prompts 1099bgiven to a model, where using AttnLRP the corresponding relevance scores of a specific neuron hast been marked in the prompts using different highlighting effects. It can be seen, that the respective neuron is activating for ‘teacher’ in unusual contexts such as sexual misconduct and inappropriate behavior. In other words, Fig. 10b shows AttnLRP Attribution for a neuron (e.g., #2207, in layer 20). The knowledge neuron is activating for ’teacher’, in unusual context such as sexual misconduct, inappropriate behavior etc. Fig. 10c shows a set of prompts 1099cgiven to a model, where using AttnLRP the corresponding relevance scores of a specific neuron hast been marked in the prompts using different highlighting effects. It can be seen, that the respective neuron is related scientific description of plants. In other words, Fig. 10c shows AttnLRP Attribution for a neuron (e.g., #922, in layer 18). The knowledge neuron seems to be activating for scientific descriptions of plants. Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most important method steps may be executed by such an apparatus. Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software or at least partially in hardware or at least partially in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are FH240201PEP-2025036018.DOCX capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable. Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed. Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier. Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier. In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer. A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory. A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein. FH240201PEP-2025036018.DOCX A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver. In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus. The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer. The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer. The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein. p References Abnar, S. and Zuidema, W. H. (2020). Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197. Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., and Lapuschkin, S. (2023). From attribution maps to human-understandable explanations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006–1019. Ali, A., Schnake, T., Eberle, O., Montavon, G., Müller, K.-R., and Wolf, L. (2022). Xai for transformers: Better explanations through conservative propagation. In International Conference on Machine Learning, pages 435–451. PMLR. FH240201PEP-2025036018.DOCX Arras, L., Osman, A., and Samek, W. (2022). Clevrxai: A benchmark dataset for the ground truth evaluation of neural network explanations. Information Fusion, 81:14–40. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7):e0130140. Balduzzi, D., Frean, M., Leary, L., Lewis, J., Ma, K. W.- D., and McWilliams, B. (2017). The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning, pages 342–350. PMLR. Bills, S., Cammarata, N., Mossing, D., Tillman, H., et al. (2023). Language models can explain neurons in language models. URL https: / / openaipublic.blob.core.windows.net / neuronexplainer / paper / index.html (accessed: 14.05.2023). Blücher, S., Vielhaben, J., and Strodthoff, N. (2024). Decoupling pixel flipping and occlusion strategy for consistent xai benchmarks. arXiv preprint arXiv:2401.06654. Caron, M., Touvron, H., Misra, I., J´egou, H., Mairal, J., Bojanowski,P., and Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 9650–9660. Chang, C.-H., Creager, E., Goldenberg, A., and Duvenaud, D. (2018). Explaining image classifiers by counterfactual generation. arXiv preprint arXiv:1807.08024. Chefer, H., Gur, S., and Wolf, L. (2021a). Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 397–406. Chefer, H., Gur, S., and Wolf, L. (2021b). Transformer interpretability beyond attention visualization. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 782–791. FH240201PEP-2025036018.DOCX Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016). Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022). Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. (2019). What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286. Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. (2022). Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493–8502. Dao, T., Fu, D., Ermon, S., Rudra, A., and R´e, C. (2022). Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359. Deb, M., Deiseroth, B., Weinbach, S., Schramowski, P., and Kersting, K. (2023). Atman: Understanding transformer predictions through memory efficient attention manipulation. arXiv preprint arXiv:2301.08110. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee. Dombrowski, A.-K., Anders, C. J., Müller, K.-R., and Kessel, P. (2022). Towards robust explanations for deep neural networks. Pattern Recognition, 121:108194. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR. Fatima, S. S., Wooldridge, M., and Jennings, N. R. (2008). A linear approximation method for the shapley value. Artificial Intelligence, 172(14):1673–1699. FH240201PEP-2025036018.DOCX Fong, R. C. and Vedaldi, A. (2017). Interpretable explanations of black boxes by meaningful perturbation. In IEEE International Conference on Computer Vision (ICCV), pages 3449–3457. Fryer, D., Str¨umke, I., and Nguyen, H. (2021). Shapley values for feature selection: The good, the bad, and the axioms. IEEE Access, 9:144352–144360. Geva, M., Caciularu, A., Wang, K., and Goldberg, Y. (2022). Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45. Geva, M., Schuster, R., Berant, J., and Levy, O. (2021). Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495. Gildenblat, J. (2020. Accessed on Dec 01, 2023). Exploring explainability for vision transformers. https: / / jacobgil.github.io / deeplearning / visiontransformer-explainability. Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D.,Turini, F., and Giannotti, F. (2018). Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820. Huang, L., Yu,W., Ma,W., Zhong,W., Feng, Z.,Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. (2023). Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463. Lundberg, S. M. and Lee, S. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30, pages 4765–4774. Maas, A., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. (2011). Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150. Mao, C., Jiang, L., Dehghani, M., Vondrick, C., Sukthankar,R., and Essa, I. (2021). Discrete representations strengthen vision transformer robustness. In International Conference on Learning Representations. FH240201PEP-2025036018.DOCX Meng, K., Bau, D., Andonian, A., and Belinkov, Y. (2022). Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372. Miglani, V., Yang, A., Markosyan, A., Garcia-Olano, D., and Kokhlikyan, N. (2023). Using captum to explain generative language models. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 165–173. Montavon, G., Binder, A., Lapuschkin, S., Samek, W., and Müller, K.-R. (2019). Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning, pages 193–209. Montavon, G., Lapuschkin, S., Binder, A., Samek, W., and Müller, K.-R. (2017). Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition, 65:211– 222. Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29. Pahde, F., Yolcu, G. Ü., Binder, A., Samek, W., and Lapuschkin, S. (2023). Optimizing explanations by network canonization and hyperparameter search. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 3818–3827. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, highperformance deep learning library. Advances in Neural Information Processing Systems, 32. Radford, A., Jozefowicz, R., and Sutskever, I. (2017). Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789. FH240201PEP-2025036018.DOCX Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). ”why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM. Samek, W., Binder, A., Montavon, G., Lapuschkin, S., and Müller, K.-R. (2017). Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660–2673. Scheepers, T. (2017). Improving the compositionality of word embeddings. Master’s thesis, Universiteit van Amsterdam, Science Park 904, Amsterdam, Netherlands. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017). Grad- cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 618–626. Shaham, U., Ivgi, M., Efrat, A., Berant, J., and Levy, O.(2023). Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196. Shrikumar, A., Greenside, P., and Kundaje, A. (2017). Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145–3153. PMLR. Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR. Smilkov, D., Thorat, N., Kim, B., Vi´egas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825. Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319–3328. PMLR. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. FH240201PEP-2025036018.DOCX Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. Voita, E., Ferrando, J., and Nalmpantis, C. (2023). Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.04827. Voita, E., Sennrich, R., and Titov, I. (2021). Analyzing the source and target contributions to predictions in neural machine translation. In 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL / IJCNLP, pages 1126–1140. Wiegreffe, S. and Pinter, Y. (2019). Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019). Huggingface’s transformers: Stateof-the-art natural language processing. arXiv preprint arXiv:1910.03771. Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In European Conference Computer Vision - ECCV 2014, pages 818–833. Zhang, B. and Sennrich, R. (2019). Root mean square layer normalization. Advances in Neural Information Processing Systems, 32. FH240201PEP-2025036018.DOCX

Claims

Claims 1. Apparatus (100) for assigning relevance scores (103) to portions (102) to be evaluated of a machine learning (ML) predictor (101) including one or more attention modules (122) and / or a softmax function (115), the apparatus (100) configured to determining the relevance scores (103) by back-propagating an initial relevance score (103) at an output (110) of the ML predictor (101) or at an internal portion of the ML predictor (101), on the basis of activations (104) of the ML predictor (101), which manifest itself in an inference performed by the ML predictor (101) in portions (123) of the ML predictor (101), which lie upstream relative to the output (110) or the internal portion along an activation direction (105) of the ML predictor (101), and include the one or more attention modules (122) and / or the softmax function (115).

2. Apparatus (100) of claim 1, wherein the ML predictor (101) comprises nodes (118) and node interconnections (119) and the apparatus (100) is configured to determine the relevance scores (103) for the nodes and / or the node interconnections of the ML predictor (101) by back propagating the initial relevance score (125) by distributing a relevance score (103) at a predetermined node (106) of the ML predictor (101) onto predecessor nodes (107) of the predetermined node (106) according to, for each of the predecessor nodes (107), a fraction associated with the respective predecessor node (107), and determining, for each predecessor node (107), the fraction associated with the respective predecessor node (107) depending on a further fraction at which the activation (104) of the respective predecessor node (107) contribute to the activation (104) of the predetermined node (106) in the inference.

3. Apparatus (100) of claim 2, configured to determine the relevance score (103) for a predetermined portion of the ML predictor (101), FH240201PEP-2025036018.DOCXcomposed of more than one node and / or node inter connection (119) of the ML predictor (101) by aggregating the relevance scores (103) of the more than one node and / or node interconnection (119) the predetermined portion is composed of.

4. Apparatus (100) of any of the previous claims, configured to sum-up, according to a summation function, the relevance scores (103) distributed from successor nodes (121) of a further predetermined node so as to obtain the relevance score (103) for the further predetermined node.

5. Apparatus (100) of any of the previous claims, configured to sum-up, according to a summation function, the relevance scores (103) distributed from successor nodes (121) of a further predetermined node so as to obtain the relevance score (103) for the further predetermined node if the further predetermined node is contained in a linear layer of the ML predictor (101).

6. Apparatus (100) of claim 5, wherein the ML predictor (101) is a Vision Transformer and the apparatus is configured to check whether the successor nodes (121) are part of a linear layer or a convolutional layer of the ML predictor (101), if the successor nodes (121) are part of a linear layer or a convolutional layer of the ML predictor (101), perform the summation over the relevance scores (103) distributed from the successor nodes (121) using a ^^-rule or a ^^ା-rule, and if the successor nodes (121) are not part of a linear layer or a convolutional layer of the ML predictor (101), perform the summation over the relevance scores (103) distributed from the successor nodes (121).

7. Apparatus (100) of any of the previous claims, configured to, in back-propagating the relevance scores (103), back-propagate the relevance scores (103) at successor nodes (121) of a softmax function (115) onto a predetermined predecessor node (107) of the softmax function (115), by forming a product between a difference between a summation over the relevance scores (103) at the successor nodes (121) of the softmax function (115), multiplied with the softmax function`s (115) output FH240201PEP-2025036018.DOCXfor a corresponding successor node (121), associated with the predetermined predecessor node (107), as obtained in the inference, and the relevance score (103) of the corresponding successor node (121), and the softmax function`s (115) input for the predetermined predecessor node (107) as obtained in the inference.

8. Apparatus (100) of any of the previous claims, configured to, in back-propagating the relevance scores (103), back-propagate the relevance scores (103) at successor nodes (121) of a softmax function (115) onto a predetermined predecessor node (107) of the softmax function (115), wherein a relevance score (103) of the predetermined predecessor node (107) is calculated usingwherein ^^^denotes the softmax function’s (115) input for the predetermined predecessor node (107) as obtained in the inference; ^^^^ା^denotes the relevance score 103 of a corresponding successor node (121);∑^ ^^ା^denotes a summation over the relevance scores ^^^^ା^(103) at the successor nodes (121) of the softmax function (115); and ^^^denotes the softmax function`s (115) output for the corresponding successor node (121).

9. Apparatus (100) of any of the previous claims configured to, in back-propagating the relevance scores (103) back-propagate the relevance scores (103) at successor nodes (121) of a dot product (114) function of an operand (111) and a further operand (112), onto a predetermined predecessor node (107) of the dot product (114), by forming a sum over summands, that are a product of an activation (104) of the predetermined predecessor node (107) as obtained in the inference, and a factor that is associated with an activation (104) of a further operand (112) of the dot product (114), and a quotient of a relevance score (103) that is associated with the successor nodes (121) as a dividend FH240201PEP-2025036018.DOCXand a sum of a product of a further factor and a sum over the products of a factor associated with the activation (104) of the operand (111) of the dot product (114) as obtained in the inference and a factor that is associated with the activation (104) of the further operand (112) of the dot product (114) as obtained in the inference and a number near zero or zero as a divisor.

10. Apparatus (100) of any of the previous claims, configured to, in back-propagating the relevance scores (103), back-propagate the relevance scores (103) at successor nodes (121) of a dot product (114) function of an operand and a further operand onto a predetermined predecessor node (107) of the dot product (114), wherein a relevance score (103) of the predetermined predecessor node (107) is calculated usingwith ^^^^ ൌ ∑^ ^^^^^^^^,,for the dot product ^^ ⋅ ^^ (114), with the operand ^^ (111) and the further operand ^^ (112) andfor the predetermined predecessor node ^^^^ (107) of the operand (111); wherein ^^^^denotes the activation (104) of an element ^^^^ of the operand (111) as obtained in the inference; ^^^^denotes the activation (104) of an element ^^^^ of the further operand (112) of the dot product (114) as obtained in the inference; and ^^ is a number near zero with a same sign as ^^^^; and ^^^^^ denotes a relevance score (103) of a successor node (121) of the dot product (114) function.

11. Apparatus (100) of any of the previous claims, FH240201PEP-2025036018.DOCXconfigured to, in back-propagating the relevance scores (103), back-propagate the relevance scores (103) at successor nodes (121) of a normalization function (117) onto a predetermined predecessor node (107) of the normalization function (117), by setting the relevance score (103) of the predetermined predecessor node (107) of the normalization function (117) to the relevance score (103) of the successor node (121).

12. Apparatus (100) of any of the previous claims, configured to, in back-propagating the relevance scores (103), back-propagate only relevance scores (103) with an absolute value greater than a certain threshold and / or with an absolute value greater than zero.

13. Apparatus (1230) for assigning relevance scores (103) to context inputs (1231) of a ML predictor (101) including one or more attention modules and / or a softmax function, the relevance scores (103) being indicative of a relevance of the context inputs (1231) for an output of the ML predictor (101) output (110) by the ML predictor (101) when fed with a query (1232) along with the context inputs (1231), wherein the apparatus (1230) comprises an apparatus (100) of any of the previous claims.

14. Apparatus (1230) of claim 13, configured to perform a highlighting of one or more portions in the context inputs (1231) depending on the relevance scores (103).

15. Apparatus (1230) of the claim 13 or 14, comprising a user interface and configured to determine a selection or subset of the context inputs (1231) or of unprocessed context inputs (1233) mapped onto the context inputs via precoding, depending on the relevance scores (103) and presenting, via the user interface, the selection or subset depending on the relevance scores (103) to the user.

16. Apparatus (1230) of claim 15, configured to perform the presentation of the selection or subset by performing automatic file opening.

17. Apparatus (1230) of any of the claims 13 to 16, configured to generate, depending on the relevance scores, a data structure which associates a specific part of an output (110) of the ML predictor (101) with a specific part of the context inputs (1231), or with a specific part of unprocessed context inputs mapped onto the context FH240201PEP-2025036018.DOCXinputs (1231) via precoding, or to present to a user association hints which associate a specific part of the output of the ML predictor (101) with a specific part of the context inputs (1231), or with specific parts of unprocessed context inputs mapped onto the context inputs (1231) via precoding.

18. Apparatus (1230) of claim 17, wherein the association hint is a citation of the specific part or the presentation of the specific part of the context inputs (1231), or the specific part of unprocessed context inputs (1231) mapped onto the context inputs (1231) via precoding is a quotation out of the context inputs (1231).

19. Apparatus (1230) of any of the claims 13 to 18, configured to sort of the context inputs (1231) depending on the relevance scores (103).

20. Apparatus (1230) of any of the claims 14 to 19, wherein the ML predictor (101) comprises an encoder (1234) which is configured to obtain a latent space representation (1233) of the context input (1231).

21. Apparatus (1120) for pruning and / or quantizing an ML predictor (101) including one or more attention modules (122) and / or a softmax function (115), comprising an apparatus (100) according to any of claims 1 to 12 to obtain relevance scores, and configured to, using the relevance scores (103), control a pruning and / or quantizing of the ML predictor (103).

22. Apparatus (1440) for controlling an activation filtering functionality (1441) of an ML predictor (101) including one or more attention modules (122) and / or a softmax function (115), comprising an apparatus (100) according to any of claims 1 to 12 to obtain relevance scores, and configured to, using the relevance scores (103), control the activation filtering functionality (1441).

23. Apparatus according to claim 22, wherein the activation filtering functionality (1441) is configured to emphasize or dampen an activation of a specific portion or specific neuron (1551) of the ML predictor during a forthcoming inference of the ML predictor, wherein the activation filtering functionality (1441) is controllable with respect to locating the specific portion or specific neuron being within the ML predictor and / or in terms of a strength of the dampening and / or the emphasizing, and the FH240201PEP-2025036018.DOCXapparatus is configured to control the activation filtering functionality (1441) with respect to the locating the specific portion or specific neuron and / or in terms of a strength of the dampening and / or the emphasizing based on the relevance scores (103), obtained by the apparatus (100) during one or more previous inferences of the ML predictor (101).

24. Apparatus (1660) for providing a certainty measure (1661) for an output (110) obtained by an inference using a ML predictor (101) including one or more attention modules (122) and / or a softmax function (115), comprising an apparatus (100) according to any of claims 1 to 12 to obtain relevance scores (103) for the inference, and configured to, using the relevance scores (103), obtain the certainty measure (1661).

25. Apparatus (1770) for auditing a ML predictor (101) including one or more attention modules (122) and / or a softmax function (115), wherein, an output of the ML predictor is composed of a plurality of portions associated with a plurality of tokens; the apparatus (1770) is configured to assign relevance scores (103) to portions (102) to be evaluated of a machine learning (ML) predictor (101) for each of a subset of the plurality of portions by back-propagating an initial relevance score (103) at the respective portion, on the basis of activations (104) of the ML predictor (101), which manifest itself in an inference performed by the ML predictor (101) in portions (123) of the ML predictor (101), which lie upstream relative to the output (110) along an activation direction (105) of the ML predictor (101), and include the one or more attention modules (122) and / or the softmax function (115); select among the plurality of tokens based on the relevance scores 103 obtained for the subset of portions and the output of the ML predictor (101).

26. Apparatus (100, 1120, 1230, 1440, 1550, 1660, 1770) of any of the previous claims, wherein the ML predictor (101) is a LLM predictor or a transformer-based model.

27. Method for assigning relevance scores (103) to portions to be evaluated of a machine learning (ML) predictor (101) including one or more attention modules (122), FH240201PEP-2025036018.DOCXthe method configured to determining the relevance scores (103) by back-propagating an initial relevance score (103) at an output (110) of the ML predictor (101) or at an internal portion of the ML predictor (101), on the basis of activations (104) of the ML predictor (101), which manifest itself in an inference performed by the ML predictor (101) in portions of the ML predictor (101), which lie upstream relative to the output (110) or the internal portion along an activation direction (105) of the ML predictor (101), and include the one or more attention modules.

28. Computer program having a program code for performing, when running on a computer, a method according to claim 26. FH240201PEP-2025036018.DOCX

Images