Token selection in a transformer neural network for efficient inference

Abstract

Certain aspects of the present disclosure provide techniques and apparatuses for processing data using a transformer neural network. The method generally includes generating, via a first attention layer of a machine learning model, a first attention map based on an input into the machine learning model; identifying, using a token prediction model, a first subset of tokens in the first attention map that are more relevant to a second attention layer of the machine learning model and a second subset of tokens in the first attention map that are less relevant to the second attention layer of the machine learning model; generating, via the second attention layer of the machine learning model, a second attention map based on the first subset of tokens in the first attention map; and generating an inference based on the second attention map and the second subset of tokens in the first attention map.

Description

[0001] Cross-references to related applications

[0002] This patent application claims priority to U.S. Patent Application No. 18 / 821,896, filed August 30, 2024, which claims the benefit and priority to U.S. Provisional Patent Application No. 63 / 633,786, filed April 14, 2024, both of which are expressly incorporated herein by reference in their entirety. Background Technology

[0003] Various aspects of this disclosure relate to neural networks, and more specifically to using neural networks to perform inference operations efficiently.

[0004] Machine learning models (such as convolutional neural networks, transformer neural networks, etc.) are used for a variety of tasks, such as object detection in visual content, segmentation of visual content, processing data of objects with different dimensions, and generating natural language responses to natural language queries. To perform these tasks, these machine learning models can be trained to perform various operations internally (e.g., mapping input data to representations in a latent space upon which inference can be performed, projecting inputs onto tokens (e.g., key, query, and value tokens in transformer neural networks), applying activation functions to data generated by the machine learning model, etc.). The complexity of these operations can range from relatively simple mathematical operations (e.g., addition, multiplication, etc.) to complex mathematical operations involving significant processor time and memory utilization. Summary of the Invention

[0005] Certain aspects of this disclosure provide a method for a processor implementation of efficient reasoning using a machine learning model. The method typically includes: generating a first attention map via a first attention layer of a machine learning model based on input to the machine learning model; using a lexical prediction model to identify a first subset of lexical units in the first attention map that are more relevant to a second attention layer of the machine learning model and a second subset of lexical units in the first attention map that are less relevant to the second attention layer of the machine learning model; generating a second attention map via the second attention layer of the machine learning model based on the first subset of lexical units in the first attention map; and generating reasoning based on the second attention map and the second subset of lexical units in the first attention map.

[0006] Certain aspects of this disclosure provide a processor implementation for training a predictive model to perform efficient inference. The method typically includes: generating multiple benchmark ground truth attention maps using a machine learning model for a set of inputs in a training dataset; training a lexical prediction model based on the training dataset to generate a predicted attention map, wherein the lexical prediction model is trained based on minimizing the difference between the predicted attention map and corresponding benchmark ground truth attention maps from the multiple benchmark ground truth attention maps, and wherein the predicted attention map includes multiple lexical units, each corresponding lexical unit being associated with a corresponding relevance score generated based on the predicted attention map; and deploying the lexical prediction model.

[0007] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0008] The following description and related figures illustrate certain exemplary features of one or more aspects. Attached Figure Description

[0009] The accompanying drawings depict exemplary features of certain aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0010] Figure 1 An example transformer neural network architecture is illustrated.

[0011] Figure 2 An example transformer neural network according to various aspects of this disclosure is illustrated, the transformer neural network including layers that selectively process lexical units generated by previous layers in the transformer neural network.

[0012] Figure 3 Pipelines for efficiently performing inference operations in a transformer neural network using a predictive model based on lexical selection, according to various aspects of this disclosure, are illustrated.

[0013] Figure 4 An example is illustrated of a pipeline for selecting lexical units for selective processing in a transformer neural network, based on various aspects of this disclosure, for training a prediction model.

[0014] Figure 5Example operations are illustrated according to various aspects of this disclosure for efficiently performing inference operations in a transformer neural network based on lexical selection.

[0015] Figure 6 Example operations for selecting lexical units for selective processing in a transformer neural network, according to various aspects of this disclosure, are illustrated.

[0016] Figure 7 Example processing systems configured to perform various aspects of this disclosure are described.

[0017] Figure 8 Example processing systems configured to perform various aspects of this disclosure are described.

[0018] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0019] This disclosure provides apparatus, methods, processing systems, and non-transitory computer-readable media for efficiently performing inference operations using transformer neural networks.

[0020] Various types of neural networks can be used to generate inferences based on input data (e.g., detecting objects, predicting the future motion of objects detected in visual content, segmenting visual content into different semantic groups, etc.), such as still images or streams of visual content (e.g., visual content captured as a series of images at a given frame rate, such as 24 frames per second, 29.97 frames per second, 60 frames per second, etc.). However, these neural networks typically process visual content on a per-frame basis, which can be a computationally expensive process, and its complexity increases with the frame size of each frame in the visual content.

[0021] Transformer neural networks (also known as "transformers"), and specifically visual transformers, have become increasingly prevalent in a wide variety of machine learning tasks. Transformer-based architectures are typically configured to generate output based on sequences of data, such as frames in a video, patches from frames or images, etc. In general, machine learning models can use any number of transformer blocks (each providing self-attention) as well as any other components, such as one or more neural network layers.

[0022] Due to the structure of these networks, using transformer neural networks to generate inference can be a computationally expensive process. Generally, the input can be projected onto multiple lexical units for processing within the transformer neural network, and each attention layer within the transformer neural network can perform an operation on each lexical unit to generate feature maps of the lexical units ingested by subsequent layers for processing. While processing each lexical unit associated with the input into the transformer neural network allows for accurate inference, this can be computationally inefficient due to the relative importance of the different lexical units generated by the layers of the transformer neural network. That is, for the first lexical unit generated by the transformer neural network... i The set of words generated by the layer, a subset of words for the first layer of the transformer neural network. i The set of terms generated by the +1 layer can be relevant. However, since each layer of a transformer neural network is generally configured to process every term input to that layer, regardless of the relevance of such terms to the inference operation performed by that layer of the neural network, computational resources may be wasted on processing terms that are unlikely to be relevant to the task using the transformer neural network. In an example where a transformer neural network is used to identify and predict the motion of objects in a scene captured in visual content, terms associated with static content (e.g., background data, non-moving objects, etc.) may be processed, even if these objects are unlikely to be associated with patterns of detecting objects in motion and predicting such motion.

[0023] Various aspects of this disclosure provide techniques for reducing the computational cost of processing input data in a transformer neural network. As discussed further in detail herein, in order to reduce the computational expenditure involved in inference operations using a transformer neural network, a predictive model can be used to predict the input data generated by the first input data in the transformer neural network. i A subset of the words generated by the layer, and this subset of words is likely to be related to the first set of words generated by the transformer neural network. i The operations performed at layer +1 are related. A subset of the predicted terms can be fed as input to the transformer neural network. i +1 layer, and can be obtained from the first layer of the transformer neural network. i The input in layer +1 omits the subset of predictions other than those for word terms, which is determined by the first... i The word units generated by the layer. The first layer of the transformer neural network. i The +1 layer can then generate the first layer with respect to the transformer neural network. i The output of the layer is reduced in size, and this output can be compared with a subset of the predictions of the third layer, excluding the word tokens. i Layer-generated word combinations to generate the first layer of the transformer neural network i The input of layer +2, which is retained in the transformer neural network's... iThe size of the input in the layers. Therefore, a variety of tasks using transformer neural networks, such as object detection or other computer vision tasks, can be accomplished with fewer computational resources, while maintaining or improving inference accuracy compared to techniques where each layer of the transformer neural network naively processes tokens without using the token selection techniques described herein. Consequently, the techniques discussed herein reduce the amount of power and / or time required to perform these tasks and / or accelerate the processing of multidimensional inputs compared to the amount of power and / or time used when outliers are not decayed in the transformer neural network.

[0024] Example Transformer Architecture

[0025] Figure 1 An example visual transformer neural network 100 is illustrated, wherein attention data is propagated through transformer encoder blocks in the neural network (e.g., to other transformer blocks in the network) to generate the output of the neural network.

[0026] like Figure 1 As illustrated, input data sample 110 is accessed by transformer encoder 140 (which is an example of a transformer block). As used herein, accessing data generally includes receiving, retrieving, requesting, or otherwise gaining access to the data. As discussed above, input data sample 110 may correspond to the input to a first transformer block of the model (e.g., raw or preprocessed input data), the output of an online transformer or other model component or block, etc. For example, input data sample 110 may correspond to a multidimensional input, a lexicalized version of the multidimensional input (which may optionally include localization embeddings and / or learnable lexicals), etc. The lexicalized version of the multidimensional input may also be referred to as a set of features generated on different parts of the multidimensional input (e.g., different spatial portions or blocks of the multidimensional input across multiple time points).

[0027] For visual data, as illustrated, input data sample 110 can be divided into multiple blocks (e.g., portions of the visual data). These blocks can have the same or different dimensions on one or both of the horizontal and vertical axes. For processing, the blocks of input data sample 110 can be linearly projected (e.g., onto a one-dimensional matrix) by projection block 120 into a linear projection 130. Within this linear projection 130, each block of the input can be mapped to a localization code that identifies its position in a multidimensional space where the visual data resides.

[0028] Generally, the converter encoder 140 includes a multi-head attention block 144 and a multilayer perceptron (MLP) 148. In the multi-head attention block 144, input data (which may be normalized by a normalization block 142 before being processed by the multi-head attention block 144) can be linearly projected (e.g., multiplied using learned parameters) into three matrices for each head of the multi-head attention block 144: a query matrix. (Also sometimes referred to as a "query representation" or simply a "query", key matrix) (Also referred to in some respects as "key representation" or simply "key"), and value matrices. (Also referred to in some contexts as "value representation" or simply "value"). For example, during training, one or more query weights, key weights, and value weights are learned based on the training data, and the query... ,key Sum It can be generated by multiplying the input data by learned weights.

[0029] In some aspects, attention matrices are then generated based on queries and keys. (Also referred to in some respects as an "attention map" or simply "attention") as the output of the transducer encoder 140. For example, the multi-head attention block 144 can compute the dot product of the query matrix and the transpose key matrix (e.g., In some respects, the multi-head attention block 144 can apply one or more operations (e.g., row-wise softmax operation) to the dot product to produce the attention matrix. In other words, the attention matrix generated by multi-head attention blocks. It can be defined as ,in This corresponds to regularization functions that can be used in transformer neural networks, such as the softmax function.

[0030] Generally speaking, given the first 100th generation of the visual transformer neural network... Input in the layer ,in Corresponding to the number of lexical units (e.g., features of image chunking), and Corresponding to the feature dimension, query ,key Sum The matrix can target each head in the multi-head attention block 144. To calculate. Generally speaking, , and ,in , and It is a linear transformation matrix. Subsequently, the... Attention Matrix of Attention Head It can be calculated based on the following equation:

[0031] This makes the first of the 100th generation of the visual transformer neural network The first layer The input in the header can be represented by the following expression:

[0032] From attention head The output of each attention head in the process can be cascaded and fed into a linear layer to generate the first attention head according to the expression. The final output of the converter layer:

[0033] in Indicates the definition of the first i The function of attention head, and This corresponds to the linear projection matrix.

[0034] Features generated by multi-head self-attention block 144 This can then be used as an attention matrix. Sum matrix The dot product is used to calculate these features. This can then be provided as input (in some respects, after normalization via normalization block 146) to multilayer perceptron 148 (e.g., a neural network or subnet) to generate an output (e.g., an attention matrix) from transformer encoder 140. The output can be used as input to subsequent transformers or other blocks in the neural network, or it can be the final result of processing the input through the neural network. For example, as... Figure 1 As illustrated, the output of the converter encoder 140 can be provided as input to a classification head 150 (which may be an MLP head or other head) for processing. The output of the classification head 150 can be a classification 160 of one or more objects in the input data sample 110.

[0035] Although Figure 1 An example is illustrated of a visual transformer neural network 100 configured to classify objects included in input data sample 110. However, it should be understood that the visual transformer neural network 100 can be trained to perform other computer vision tasks, such as depth estimation, motion prediction, etc., and may include different components suitable for performing such tasks. Furthermore, although Figure 1An example of using a multilayer perceptron within converter encoder 140 to generate the output of converter encoder 140 is illustrated, but it should be appreciated that any kind of feedforward block can be used to generate the output of converter encoder 140. Furthermore, it should be understood that converter encoder 140 can utilize any suitable architecture, and other examples of converter encoder 140 are conceivable.

[0036] Generally, within a transformer neural network (e.g., a visual transformer neural network 100), attention can be computed for each word provided as input to the transformer neural network, relative to all other words provided as input. Therefore, the computational expenditure and memory cost involved in processing the input in a transformer neural network are typically relative to the number of words in the input. Scaling quadratically (e.g., making the computational and memory costs of processing inputs in a transformer neural network proportional to...) (For scaling purposes). However, the terms within the input may have different levels of importance for the reasoning process performed by a transformer neural network (such as a visual transformer neural network 100). For example, within an image, different blocks (terms) may carry different information, where reasoning can be generated quickly for blocks with little semantic information (e.g., blocks depicting the sky) (e.g., using a small number of layers of the neural network), and where reasoning is performed using a large number of layers for blocks with a lot of semantic information (e.g., blocks depicting buildings, vegetation, road surfaces, and / or other objects that may be relevant to a given task such as object recognition in autonomous driving applications).

[0037] To reduce the computational and memory costs involved in processing inputs in a transformer neural network, various techniques can be used to reduce the number of tokens processed by the layers within the transformer neural network. In some examples, binary matching can be performed on tokens within each layer of the transformer neural network, where the first... r Each match is processed. In such examples, matched terms can be merged together and tracked, which can result in each term being processed through each layer of the transformer neural network. Furthermore, because semantically similar terms are merged, it may be impossible to learn or detect distinguishing features in the blocks corresponding to these merged terms, and the coarsness of the feature maps generated by the transformer neural network can increase with each layer in the neural network.

[0038] In other examples, terms in the input or feature map can be downsampled to reduce the number of terms processed by subsequent layers in the neural network. Generally, terms can be merged based on spatial proximity, allowing merging without the computational complexity involved in binary matching. However, since adjacent image blocks may not have the same or similar semantic meanings, downsampling techniques can effectively reduce the detail of the feature map and decrease its spatial resolution, which can negatively impact the accuracy of inference generated by the transformer neural network.

[0039] In other examples, lexical units can be scored based on their importance to the final classification task. Based on these scores, lexical units with relatively high importance to the classification task can be retained, while those with relatively low importance can be discarded (or downsampled). Therefore, the number of lexical units processed by the layers of the transformer neural network can decrease with each successive layer. However, since information is discarded at each layer of the transformer neural network, score-based downsampling techniques may not be recommended for tasks more complex than classification; for example, these downsampling techniques may not be recommended for intensive prediction tasks such as depth estimation or semantic segmentation.

[0040] Certain aspects of this disclosure provide techniques for reducing the computational cost of inference tasks in transformer neural networks by leveraging the semantic importance of lexical units, while allowing semantic information to be preserved throughout the transformer neural network. In doing so, certain aspects of this disclosure reduce redundant computation by bypassing the processing of lexical units with lower importance for a given task within a layer of the neural network. Furthermore, certain aspects of this disclosure maintain inference accuracy by combining these bypassed lexical units with feature maps generated by the layers of the neural network based on semantically important lexical units.

[0041] Using lexical selection for efficient inference in transformer neural networks

[0042] Figure 2 An example transformer neural network 200 according to various aspects of the present disclosure is illustrated, the transformer neural network including layers that selectively process lexical units generated by previous layers in the transformer neural network.

[0043] As illustrated, the transformer neural network 200 comprises multiple layers alternating between standard transformer layers and selective lexical attention layers. Each pair of standard transformer layers and selective lexical attention layers can be organized into layer blocks 202, 204 (and other layer blocks as well). Figure 1 (Not illustrated in the text). In standard transformer layers (such as standard transformer layer 210 in layer block 202 or standard transformer layer 220 in layer block 204), lexicalization of input is performed. Can be processed to generate queries ,key Sum These matrices , and By converting the input into a lexical form It is generated by multiplying by the learned weights of the transformer neural network 200, as described above relative to... Figure 1 The output is discussed. Representation and Lexical Input The associated feature map can include multiple lexical units.

[0044] To reduce the computational overhead involved in processing data in the converter neural network, the outputs of standard converter layers 210 and 220... , These can be fed into attention map prediction blocks 212 and 222 and lexical selection blocks 214 and 224 of layers 202 and 204, respectively, for processing. Attention map prediction blocks 212 and 222 generally include prediction models that score or rank lexical assignments (or otherwise prioritize lexical assignments) in the input to the selective lexical attention layers in the transformer neural network 200 (e.g., selective attention transformer layers 216 and 226 of layers 202 and 204, respectively).

[0045] The attention maps generated by the previous standard transformer layers 210 and 220 are based on the word selection blocks 214 and 224. , To assign the outputs of standard converter layers 210 and 220 , The score for each corresponding word in (which is used as input to the selective attention transformer layer of transformer neural network 200) is generally a score representing the importance of that corresponding word to the output of selective attention transformer layers 216, 226. Based on the corresponding attention maps generated by the previous standard transformer layers 210, 220 from attention map prediction blocks 212, 222. , for , The score (ranking or other priority) assigned to each lexical element in the lexical selection blocks 214 and 224 can be identified ( i They are respectively , The first subset of words that are more relevant to the outputs (e.g., feature maps) generated by the selective attention transformer layers 216 and 226, respectively, and ( ii ) (In layer 202) and (In layer 204) a second subset of lexical units that are less relevant to the output of the selective attention transformer layers 216, 226. The first subset of lexical units may include, for example, lexical units with a relevance score or other predicted score exceeding a threshold (which may be learned or defined a priori). (In layer 202) and (In layer 204) the previous k lexical units (of which) k (Whether it is learned or defined a priori), etc. The second subset of lexical units includes lexical units that may include... (In layer 202) and (In layer 204) Lexicons that are not included in the first subset of lexicons.

[0046] The first subset of lexical units (marked in layer 202 as) And marked as in layer block 204 This can also be referred to as the set of "lexical units for attention processing". The second subset of lexical units (marked in layer block 202 as...) And marked as in layer block 204 This can be called the set of "direct-access lexical units". The first subset of lexical units. , These can be fed into selective attention transformer layers 216 and 226 for processing to generate intermediate outputs, each containing multiple output terms. , The intermediate output of layer block 202 And the intermediate output of layer block 204 Can be associated with the second subset of lexical units respectively , Combined to generate the outputs of selective lexical attention layers 218 and 228, respectively. , By doing this, in layer block 202 , and With layer 204 , and They can have the same dimension (e.g., such that any ),in B Indicates batch size, L This indicates the number of lexical units on which attention is calculated, and D express LThe dimension of the feature vector for each word in the lexical unit is determined, and therefore, each layer in the transformer neural network can generate a feature map based on an input of the same size. That is, while a subset of words generated by standard transformer layers 210, 220 can be processed in the associated selective attention transformer layers 216, 226 in blocks 202, 204, the next standard transformer layer can still process pass-through words not processed in selective attention transformer layers 216, 226. In doing so, words considered less important can be discarded in selective attention transformer layers 216, 226 but can be re-added for subsequent processing (e.g., through the standard transformer layers in the next block), because such words may include information related to another layer of the transformer neural network 200. Therefore, the reduction in inference accuracy caused by the lexical downsampling discussed above can be minimized (or at least reduced) because the aspects of this disclosure do not permanently discard pass-through lexical units, allowing subsequent layers of the transformer neural network 200 to operate on the entire set of data rather than a downsampled set of lexical units that discard information from the entire set of data.

[0047] Figure 3 Pipeline 300 according to various aspects of this disclosure is illustrated for efficiently performing inference operations in a transformer neural network (e.g., transformer neural network 200) using a prediction model based on lexical selection.

[0048] As illustrated, pipeline 300 includes a converter neural network (e.g., Figure 2 The illustrated converter neural network 200) is in its first... i Layer, prediction model 306, and the first layer used to identify the transformer neural network. i The word selection modules 3121, 3122, ..., 312 in layer +1 n (Collectively referred to as "lexical selection module 312"). Generally speaking, the first... i Multiple attention graphs 3021, 3022, ..., 302 are generated in the layer. n (Collectively referred to as "Attention Map 302" and labeled as "...) "),in n Corresponding to the first step of the transformer neural network i The number of attention heads included in the layer. Attention graph 302 can be cascaded into the first layer of a transformer neural network. i Attention diagram 304 of layer combination (referred to as " This information is then input into the prediction model 306 for processing. Generally, each attention map 302 can be of dimension 1. A square matrix, where Corresponding to the input in the layer of the transformer neural network The number of lexical units included.

[0049] Prediction model 306 can be a machine learning model, which is trained based on the first transformer neural network. i Layer n The attention map 302 generated by the head is used to predict the first step of the transformer neural network. i The output of layer +1. The prediction model 306 can be, for example, a convolutional neural network comprising one or more convolutional layers and a non-linear layer (such as a softmax layer or a log-softmax layer), which generates a probability score for each word included in the attention map. Generally, each prediction attention map is 3081, 3082, ..., 308. n (Collectively referred to as "Predicted Attention Map 308") can be dimensional like Attention Map 302. A square matrix, where This corresponds to the number of tokens included in the input to the layers in the transformer neural network. .

[0050] The first one based on the transformer neural network i The sum of values ​​on the columns of the predicted attention map 308 at layer +1 is used to generate a predicted score (or ranking or other priority) for each word in the predicted attention map, which is generated by the prediction model 306 for each attention head in the layers of the transformer neural network. The predicted score for each word can be included in a one-dimensional scoring matrix 3101, 3102, ..., 310 associated with the corresponding predicted attention map 308. n (Collectively referred to as "scoring matrix 310"), as discussed, the corresponding predicted attention map can be the predicted attention map of a specific attention head in the attention layer of the transformer neural network. For each attention head, it can be in the word selection blocks 3121, 3122, ..., 312 n (collectively referred to as "lexical selection block 312") selects the top lexical scores based on the score matrix 310 for the corresponding attention head. k 1 lexical unit for inclusion in the first subset of lexical units In, and the remaining word elements (e.g., (a lexical unit) can be included in the second subset of lexical units. In the middle. Then, word elements. It can be input into the layers of the transformer neural network for processing (e.g., via...). Figure 2 The illustrated selective attention transformer layers 216, 226), and the output of the transformer neural network layer can be compared with... Combine to generate output As discussed above.

[0051] Figure 4 An example is illustrated of selecting lexical units for use in training a prediction model 406 according to various aspects of this disclosure (e.g., in a transformer neural network). Figure 2 The pipeline 400 that performs selective processing in the illustrated converter neural network 200.

[0052] As illustrated, the first step can be based on the transformer neural network. i The baseline ground truth attention map of +1 layer is used to train the prediction model 406. This is done in the transformer neural network... i The baseline truth attention maps 4021, 4022, ..., 402 generated in the layers n (collectively referred to as "benchmark truth attention graph 402") can be mapped to the first... i The baseline truth attention graphs for layer +1 are 4101, 4102, ..., 410. n (Collectively referred to as "benchmark truth attention map 410"). The prediction model 406 can be trained as the first based on the transformer neural network. i The output of the layer is used to predict the first layer of the transformer neural network. i The output of layer +1 (e.g., the output of the transformer neural network) i Layer n The generated reference truth attention map 402 can be combined into a cascaded reference truth attention map 404, such as... Figure 4 (As illustrated). Doing so, in the given transformer neural network's... i Given the input of a specific output layer, the prediction model 406 can be trained to make the transformer neural network's first... i The loss between the baseline ground truth attention map and the predicted attention map at layer +1 is minimized or at least reduced. In some respects, the loss functions 4121, 4122, ..., 412... n (collectively referred to as "loss function 412") can be the loss function in the given transformer neural network. i In the case of the output of the layer, the transformer of the neural network is the first i +1 layer baseline attention map 410 and predicted attention maps 4081, 4082, ..., 408 n The Körbek-Leibler (KL) divergence loss between (collectively referred to as “Predicted Attention Map 408”).

[0053] Although Figures 2 to 4 An example is given that in the first... i Using a prediction model to predict the output of the network in the case of the transformer neural network i +1 related terms for the transformer layer, but it should be recognized that the transformer neural network can include any number of selective attention transformer layers. For example, Figure 2An architecture in which standard transformer layers and selective transformer layers alternate is illustrated (e.g., such that even layers correspond to standard transformer layers and odd layers correspond to selective attention transformer layers, and vice versa). However, any (non-zero) number of consecutive standard transformer layers will be followed by selective attention transformer layers, and any (non-zero) number of standard transformer layers may be followed by selective transformer layers.

[0054] As discussed, the techniques described in this paper generally provide a significant reduction in computational overhead when processing inputs in a transformer neural network. For example, in the first... i 80% of the words from the layer's output are fed as the first word into the transformer neural network. i Given the input in layer +1, the transformer neural network's... i The computational cost and memory utilization of layer +1 can be reduced by 30%. This is comparable to processing data from the first layer of a transformer neural network. i The first in +1 layer i Such efficiency allows for significantly reduced time and computational overhead in performing inference operations within transformer neural networks compared to the time and computational overhead involved in all the terms in the layer's output. Furthermore, the techniques discussed in this paper can provide inference accuracy similar to those used in reducing the amount of data processed within layers of transformer neural networks by not employing term selection.

[0055] Figure 5 Examples of methods for using lexical selection in transformer neural networks (e.g., based on various aspects of this disclosure) are illustrated. Figure 2 The illustrated converter neural network 200, Figure 3 Example operation 500 performs inference operations in the illustrated lexical prediction model, etc. Operation 500 may, for example, be performed by a computing system (such as a user equipment (UE), smartphone, tablet computer, autonomous vehicle, edge device, or other computing system) on which a transformer neural network (e.g., transformer neural network 200) is deployed for processing nonlinear data. Figure 7 The processing system 700, as illustrated and described in further detail below, performs the operation.

[0056] As illustrated, operation 500 begins at box 510, where a first attention map is generated via the first attention layer of the machine learning model based on the input to the machine learning model. .

[0057] At box 520, operation 500 continues to use the lexical prediction model (e.g., Figure 3 The illustrated prediction model 306) identifies the first subset and the second subset of lexical units based on a first attention map. The first subset of lexical units... This can be a subset of lexical units that are more relevant to the output of the second attention layer of the machine learning model (and can be processed by the second attention layer of the machine learning model). It can be a lexical that is not very relevant to the output of the second attention layer (and therefore can be bypassed by the second attention layer of the machine learning model).

[0058] In some respects, to identify a first subset of lexical units that are more relevant to the second attention layer of the machine learning model, a relevance score for each lexical unit can be calculated based on predictions of the second attention map generated by the lexical unit prediction model. The lexical unit with the highest relevance score can then be selected. k Each word is used to include a word in the first subset of words. k It can be selected based on the attention ratio, which is defined as the percentage of total input lexical units to be processed by the second attention layer of the machine learning model.

[0059] At box 530, operation 500 continues to generate a second attention map based on a first subset of lexical units via the second attention layer of the machine learning model.

[0060] At box 540, operation 500 continues to generate inference based on the second attention map and a second subset of lexical units. In some aspects, for this purpose, the second attention map and the second subset of lexical units from the first attention map can be concatenated into a combined attention map. This combined attention map can be fed into another layer of the machine learning model for processing. In some aspects, such as when the second attention layer of the machine learning model is the last attention layer of the machine learning model, the combined attention map can be fed into one or more other layers of the machine learning model (e.g., non-linear layers, such as softmax layers, multilayer perceptrons, etc.) for generating inference. Inference may include, for example, the identification of objects in the visual input data, depth prediction for different objects depicted in the visual input data, semantic segmentation of the visual input data into different regions corresponding to different classes of objects, etc.

[0061] In some aspects, operation 500 also includes generating an attention map by a prior attention layer of the machine learning model based on the input to the machine learning model. The input to the first attention layer of the machine learning model may include the attention map generated by the prior attention layer.

[0062] In some aspects, operation 500 may also include concatenating the second attention map and a second subset of lexical units into a cascaded attention map. Using a third attention layer of a machine learning model, the third attention map can be generated based on the cascaded attention map. Using a lexical prediction model, the third subset of lexical units more relevant to the fourth attention layer of the machine learning model and the fourth subset of lexical units less relevant to the fourth attention layer of the machine learning model can be identified based on the third attention map. The fourth attention map can be generated based on the third subset of lexical units via the fourth attention layer of the machine learning model. Inference can be further generated based on the fourth attention map and the fourth subset of lexical units.

[0063] In some aspects, the first attention map and the second attention map include attention maps for a first attention head in the machine learning model. In some aspects, the lexical prediction model includes a model specific to the first attention head. In some aspects, identifying a first subset and a second subset of lexical units involves predicting relevant lexical units based on a concatenation of the attention map for the first attention head and attention maps for one or more additional attention heads in the machine learning model.

[0064] In some respects, the lexical prediction model includes a convolutional model configured to generate a predicted attention map for a second attention layer of the machine learning model based on the input of a first attention map generated by a first attention layer of the machine learning model.

[0065] In some respects, lexical prediction models include models configured to generate normalized attention maps for multiple attention heads in a machine learning model.

[0066] Figure 6 An example operation 600 is illustrated, according to various aspects of this disclosure, for selecting lexical units for selective processing in a transformer neural network during training a prediction model. Operation 600 may be, for example, performed by training one or more machine learning models (e.g., Figure 2 The illustrated converter neural network 200, Figure 3 The exemplified lexical prediction model, etc.) computing system (such as server computers, clusters of physical or cloud computing instances, or other computing systems (e.g., such as Figure 8 The processing system 800, as illustrated and described in further detail below, performs the operation.

[0067] Operation 600 begins at box 610, where a machine learning model is used to generate multiple benchmark ground truth attention maps for a set of inputs in the training dataset. The multiple benchmark ground truth attention maps may include attention maps generated by one or more layers in a trained transformer neural network. In some aspects, the inputs in the training dataset may include lexicalized versions of the inputs to the transformer neural network or attention maps (or feature maps) generated by layers within the transformer neural network, etc.

[0068] At box 620, operation 600 continues to train a lexical prediction model based on the training dataset to generate a predicted attention map. Generally, the lexical prediction model can be trained based on minimizing the differences between the predicted attention map and corresponding ground truth attention maps from multiple ground truth attention maps. The predicted attention map may include multiple lexical units, where each corresponding lexical unit is associated with a corresponding relevance score generated based on the predicted attention map.

[0069] For example, as discussed, the attention map for the predictions generated by the lexical prediction model can be... A matrix is ​​used to represent the relevance of a given term to other terms in the attention graph. This relevance can be defined, for example, by a trained softmax function or other nonlinear function that generates probability values ​​representing the relevance of the input to a given output. Since each column in the attention graph represents a specific term, the relevance score for each term can be calculated based on the sum of the elements in the corresponding column for that term.

[0070] In some respects, training a lexical prediction model involves training the lexical prediction model based on minimizing the Kolb-Leibler (KL) divergence loss between the predicted attention map and the corresponding ground truth attention map.

[0071] At box 630, operation 600 continues to deploy the lexical prediction model.

[0072] In some respects, the word prediction model includes a convolutional model trained to generate a predicted attention map for a second attention layer of the machine learning model based on the input of a first attention map generated by a first attention layer of the machine learning model.

[0073] In some respects, lexical prediction models include models trained to generate normalized attention maps for multiple attention heads in a machine learning model.

[0074] In some respects, machine learning models include frozen transformer neural networks.

[0075] Example processing system for efficient inference in transformer neural networks using lexical selection

[0076] Figure 7 An example processing system 700 is depicted, configured to perform various aspects of this disclosure, including, for example, relative to... Figure 2 , Figure 3 and Figure 5 The techniques and methods described. In some aspects, the processing system 700 can use trained transformer-based machine learning models (such as...) Figure 2The transformer neural network 200 performs the inference operations. Although it is depicted as a single system for clarity of concept, in at least some respects, as discussed above, the operations described below with respect to the processing system 700 can be distributed across any number of devices.

[0077] The processing system 700 includes a central processing unit (CPU) 702, which in some examples may be a multi-core CPU. Instructions executed at the CPU 702 may be loaded, for example, from program memory associated with the CPU 702 or from a partition of memory 724.

[0078] The processing system 700 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 704, a digital signal processor (DSP) 706, a neural processing unit (NPU) 708, a multimedia processing unit 710, and a wireless connectivity component 712.

[0079] NPUs (such as the NPU 708) are typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0080] NPUs (such as the NPU 708) are configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip (such as a system-on-a-chip (SoC)), while in other examples, an NPU may be part of a dedicated neural network accelerator.

[0081] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0082] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0083] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data slices as input and rapidly process that new data through an already trained model to generate model outputs (e.g., inference).

[0084] In some implementations, the NPU 708 is part of one or more of the CPU 702, GPU 704, and / or DSP 706.

[0085] In some examples, the wireless connectivity component 712 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless transmission standards. The wireless connectivity component 712 is further coupled to one or more antennas 714.

[0086] The processing system 700 may also include one or more sensor processing units 716 associated with any type of sensor, one or more image signal processors (ISPs) 718 associated with any type of image sensor, and / or a navigation component 720, which may include a satellite-based positioning system component (e.g., GPS or GLONASS) and an inertial positioning system component.

[0087] The processing system 700 may also include one or more input and / or output devices 722, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0088] In some examples, one or more processors in the processing system 700 may be based on the ARM or RISC-V instruction set.

[0089] The processing system 700 also includes a memory 724, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 724 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 700.

[0090] Specifically, in this example, the memory 724 includes an attention map component 724A, a lexical identification component 724B, an inference generation component 724C, and a transformer neural network 724D. Although for the sake of conceptual clarity... Figure 7 The components are depicted as discrete components, but in various respects, the illustrated components (and other components not depicted) can be implemented together or separately.

[0091] Generally speaking, the processing system 700 and / or its components can be configured to perform the methods described herein.

[0092] It is worth noting that, in other aspects, such as when the processing system 700 is a server computer, various aspects of the processing system 700 can be omitted. For example, the multimedia processing unit 710, wireless connectivity component 712, sensor processing unit 716, ISP 718, and / or navigation component 720 can be omitted in other aspects. Furthermore, the various aspects of the processing system 700 can be distributed among multiple devices.

[0093] Figure 8 An example processing system 800 is depicted, configured to perform various aspects of this disclosure, including, for example, relative to Figure 4 and Figure 6 The techniques and methods described. In some aspects, the processing system 800 can be trained, implemented, or provided for predicting transformer neural networks. i The first layer of the neural network with the transformer i Machine learning models of word relevance in the output of +1 layer, such as in Figure 4 As illustrated in the example. Although depicted as a single system for clarity of concept, in at least some respects, as discussed above, the operations described below with respect to processing system 800 can be distributed across any number of devices.

[0094] The processing system 800 includes a central processing unit (CPU) 802, which in some examples may be a multi-core CPU. Instructions executed at the CPU 802 may be loaded, for example, from program memory associated with the CPU 802 or from a partition of memory 824.

[0095] The processing system 800 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, a multimedia processing unit 810, and a wireless connectivity component 812.

[0096] In some implementations, the NPU 808 is part of one or more of the CPU 802, GPU 804, and / or DSP 806.

[0097] In some examples, the wireless connectivity component 812 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless transmission standards. The wireless connectivity component 812 is further coupled to one or more antennas 814.

[0098] The processing system 800 may also include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor, and / or a navigation component 820, which may include a satellite-based positioning system component (e.g., GPS or GLONASS) and an inertial positioning system component.

[0099] The processing system 800 may also include one or more input and / or output devices 822, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0100] In some examples, one or more processors in the processing system 800 may be based on the ARM or RISC-V instruction set.

[0101] The processing system 800 also includes a memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 824 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 800.

[0102] Specifically, in this example, the memory 824 includes an attention map generation component 824A, a model training component 824B, a model deployment component 824C, and a transformer neural network 824D. Although for the sake of conceptual clarity... Figure 8 The components are depicted as discrete components, but in various respects, the illustrated components (and other components not depicted) can be implemented together or separately.

[0103] Generally speaking, the processing system 800 and / or its components can be configured to perform the methods described herein.

[0104] It is worth noting that, in other aspects, such as when the processing system 800 is a server computer, various aspects of the processing system 800 may be omitted. For example, the multimedia processing unit 810, the wireless connectivity component 812, the sensor processing unit 816, the ISP 818, and / or the navigation component 820 may be omitted in other aspects. Furthermore, the various aspects of the processing system 800 may be distributed among multiple devices.

[0105] Example Terms

[0106] Specific details of various aspects of this disclosure are described in the following numbered clauses: Clause 1: A processor implementation method for machine learning, the method comprising: generating a first attention map based on input to the machine learning model via a first attention layer of the machine learning model; identifying a first subset of lexical units more relevant to a second attention layer of the machine learning model and a second subset of lexical units less relevant to the second attention layer of the machine learning model based on the first attention map using a lexical prediction model; generating a second attention map based on the first subset of lexical units via the second attention layer of the machine learning model; and generating inference based on the second attention map and the second subset of lexical units.

[0107] Clause 2: The method according to Clause 1 further comprises: generating an attention map by a prior attention layer of the machine learning model based on the input to the machine learning model, wherein the input to the first attention layer of the machine learning model includes the attention map generated by the prior attention layer.

[0108] Clause 3: The method according to Clause 1 or 2, wherein generating the inference based on the second attention map and the second subset of lexical units comprises: concatenating the second attention map and the second subset of lexical units into a concatenated attention map; and generating the inference based on the concatenated attention map.

[0109] Clause 4: The method according to any one of Clauses 1 to 3, the method further comprising: concatenating the second attention map and the second subset of lexical units into a concatenated attention map; generating a third attention map based on the concatenated attention map via a third attention layer of the machine learning model; using the lexical prediction model based on the first attention map to identify a third subset of lexical units more relevant to the fourth attention layer of the machine learning model and a fourth subset of lexical units less relevant to the fourth attention layer of the machine learning model; and generating a fourth attention map based on the third subset of lexical units via the fourth attention layer of the machine learning model, wherein the inference is further generated based on the fourth attention map and the fourth subset of lexical units.

[0110] Clause 5: The method according to any one of Clauses 1 to 4, wherein identifying the first subset of lexical units more relevant to the second attention layer of the machine learning model comprises: calculating a relevance score for each lexical unit based on predictions of the second attention map generated by the lexical unit prediction model; and selecting k lexical units with the highest relevance scores for inclusion in the first subset of lexical units.

[0111] Clause 6: The method described in Clause 5, wherein k is selected based on an attention ratio, which is defined as the percentage of total input lexical units to be processed by the second attention layer of the machine learning model.

[0112] Clause 7: The method according to any one of Clauses 1 to 6, wherein the first attention map and the second attention map comprise attention maps for a first attention head in the machine learning model, and wherein the lexical prediction model comprises a model specific to the first attention head.

[0113] Clause 8: The method according to Clause 7, wherein the first subset of identifying lexical units and the second subset of identifying lexical units comprises predicting relevant lexical units based on a concatenation of the attention map for the first attention head and attention maps for one or more additional attention heads in the machine learning model.

[0114] Clause 9: The method according to any one of Clauses 1 to 8, wherein the lexical prediction model includes a convolutional model configured to generate a predicted attention map for a second attention layer of the machine learning model based on an input of a first attention map generated by the first attention layer of the machine learning model.

[0115] Clause 10: The method according to any one of Clauses 1 to 9, wherein the lexical prediction model includes a model configured to generate normalized attention maps for multiple attention heads in the machine learning model.

[0116] Item 11: A processor implementation method for machine learning, the processor implementation comprising: using a machine learning model to generate a plurality of benchmark ground truth attention maps for a set of inputs in a training dataset; training a lexical prediction model based on the training dataset to generate a predicted attention map, wherein the lexical prediction model is trained based on minimizing the difference between the predicted attention map and a corresponding benchmark ground truth attention map from the plurality of benchmark ground truth attention maps, and wherein the predicted attention map includes a plurality of lexical units, each corresponding lexical unit being associated with a corresponding relevance score generated based on the predicted attention map; and deploying the lexical prediction model.

[0117] Clause 12: The method according to Clause 11, wherein the lexical prediction model includes a convolutional model trained to generate the predicted attention map for a second attention layer of the machine learning model based on input of a first attention map generated by a first attention layer of the machine learning model.

[0118] Clause 13: The method according to Clause 11 or 12, wherein the lexical prediction model includes a model trained to generate normalized attention maps for multiple attention heads in the machine learning model.

[0119] Clause 14: The method according to any one of Clauses 11 to 13, wherein training the lexical prediction model comprises training the lexical prediction model based on minimizing the Kolb-Leibler (KL) divergence loss between the predicted attention map and the corresponding ground truth attention map.

[0120] Clause 15: The method according to any one of Clauses 11 to 14, wherein the machine learning model comprises a frozen transformer neural network.

[0121] Clause 16: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of Clauses 1 to 15.

[0122] Clause 17: A processing system comprising components for performing the method according to any one of Clauses 1 to 15.

[0123] Clause 18: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any one of Clauses 1 to 15.

[0124] Clause 19: A computer program product embodied on a computer-readable storage medium, the computer program product comprising code for performing a method according to any one of Clauses 1 to 15.

[0125] Additional Notes

[0126] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined in others. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of the disclosure herein may be embodied by one or more elements of the claims.

[0127] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as preferred or superior to other aspects.

[0128] As used in this article, the phrase “at least one of” in a list of items refers to any combination of these items, including a single member. For example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0129] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0130] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0131] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No element of any claim should be interpreted in accordance with 35 USC § 112(f) unless the element is expressly referred to by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will be known later, are expressly incorporated herein by reference and are intended to be covered by the claims. Moreover, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A processing system for machine learning, the processing system comprising: At least one memory, wherein executable instructions are stored in the at least one memory; and One or more processors, the one or more processors being configured to execute the executable instructions, so that the processing system: A first attention map is generated by the first attention layer of the machine learning model based on the input to the machine learning model; The lexical prediction model is used to identify a first subset of lexical units that are more relevant to the second attention layer of the machine learning model and a second subset of lexical units that are less relevant to the second attention layer of the machine learning model, based on the first attention map. A second attention map is generated by the second attention layer of the machine learning model based on the first subset of lexical units; as well as Reasoning is generated based on the second attention map and the second subset of lexical units.

2. The processing system of claim 1, wherein the one or more processors are further configured to generate an attention map by a prior attention layer of the machine learning model based on the input to the machine learning model, wherein the input to the first attention layer of the machine learning model includes the attention map generated by the prior attention layer.

3. The processing system of claim 1, wherein, in order to generate the inference based on the second attention map and the second subset of lexical units, the one or more processors are configured to cause the processing system to: Concatenate the second attention map and the second subset of lexical units into a concatenated attention map; and The inference is generated based on the cascaded attention map.

4. The processing system of claim 1, wherein the one or more processors are further configured to cause the processing system to: The second attention map and the second subset of the word are concatenated into a concatenated attention map; A third attention map is generated based on the cascaded attention map via the third attention layer of the machine learning model; The lexical prediction model is used to identify a third subset of lexical units that are more relevant to the fourth attention layer of the machine learning model and a fourth subset of lexical units that are less relevant to the fourth attention layer of the machine learning model, based on the first attention map. as well as The fourth attention map is generated by the fourth attention layer of the machine learning model based on the third subset of lexical units, wherein the inference is further generated based on the fourth attention map and the fourth subset of lexical units.

5. The processing system of claim 1, wherein, in order to identify the first subset of lexical units more relevant to the second attention layer of the machine learning model, the one or more processors are configured to cause the processing system to: The relevance score of each lexical is calculated based on the prediction of the second attention map generated by the lexical prediction model; and Select the one with the highest relevance score. k A set of lexical units are provided for inclusion in the first subset of lexical units.

6. The processing system according to claim 5, wherein k It is selected based on the attention ratio, which is defined as the percentage of total input lexical units to be processed by the second attention layer of the machine learning model.

7. The processing system of claim 1, wherein the first attention map and the second attention map comprise attention maps for a first attention head in the machine learning model, and wherein the lexical prediction model comprises a model specific to the first attention head.

8. The processing system of claim 7, wherein, in order to identify the first subset of lexical units and the second subset of lexical units, the one or more processors are configured to cause the processing system to predict relevant lexical units based on a cascade of the attention map for the first attention head and the attention maps for one or more additional attention heads in the machine learning model.

9. The processing system of claim 1, wherein the lexical prediction model comprises a convolutional model configured to generate a predicted attention map for a second attention layer of the machine learning model based on an input of a first attention map generated by the first attention layer of the machine learning model.

10. The processing system of claim 1, wherein the lexical prediction model includes a model configured to generate normalized attention maps for a plurality of attention heads in the machine learning model.

11. A processing system for machine learning, the processing system comprising: At least one memory, wherein executable instructions are stored in the at least one memory; and One or more processors, the one or more processors being configured to execute the executable instructions, so that the processing system: Use machine learning models to generate multiple benchmark ground value attention maps for a set of inputs in the training dataset; A word prediction model is trained based on the training dataset to generate a predicted attention map, wherein the word prediction model is trained based on minimizing the difference between the predicted attention map and the corresponding benchmark ground attention map from the plurality of benchmark ground attention maps, and wherein the predicted attention map includes a plurality of words, each corresponding word being associated with a corresponding relevance score generated based on the predicted attention map. as well as Deploy the lexical prediction model.

12. The processing system of claim 11, wherein the lexical prediction model comprises a convolutional model trained to generate the predicted attention map for a second attention layer of the machine learning model based on input of a first attention map generated by a first attention layer of the machine learning model.

13. The processing system of claim 11, wherein the lexical prediction model comprises a model trained to generate normalized attention maps for multiple attention heads in the machine learning model.

14. The processing system of claim 11, wherein, in order to train the lexical prediction model, the one or more processors are configured to train the lexical prediction model based on minimizing the Kolb-Leibler (KL) divergence loss between the predicted attention map and the corresponding ground truth attention map.

15. The processing system of claim 11, wherein the machine learning model comprises a frozen transformer neural network.

16. A processor-implemented method for machine learning, the method comprising: A first attention map is generated by the first attention layer of the machine learning model based on the input to the machine learning model; The lexical prediction model is used to identify a first subset of lexical units that are more relevant to the second attention layer of the machine learning model and a second subset of lexical units that are less relevant to the second attention layer of the machine learning model, based on the first attention map. A second attention map is generated by the second attention layer of the machine learning model based on the first subset of lexical units; as well as Reasoning is generated based on the second attention map and the second subset of lexical units.

17. The method according to claim 16, further comprising: An attention map is generated by a prior attention layer of the machine learning model based on the input to the machine learning model, wherein the input to the first attention layer of the machine learning model includes the attention map generated by the prior attention layer.

18. The method of claim 16, wherein generating the inference based on the second attention map and the second subset of lexical units comprises: The second attention map and the second subset of the word are concatenated into a concatenated attention map; as well as The inference is generated based on the cascaded attention map.

19. The method of claim 16, further comprising: The second attention map and the second subset of the word are concatenated into a concatenated attention map; A third attention map is generated based on the cascaded attention map via the third attention layer of the machine learning model; The lexical prediction model is used to identify a third subset of lexical units that are more relevant to the fourth attention layer of the machine learning model and a fourth subset of lexical units that are less relevant to the fourth attention layer of the machine learning model, based on the first attention map. as well as The fourth attention map is generated by the fourth attention layer of the machine learning model based on the third subset of lexical units, wherein the inference is further generated based on the fourth attention map and the fourth subset of lexical units.

20. The method of claim 16, wherein the first subset of lexical units more relevant to the second attention layer of the machine learning model comprises: The relevance score of each lexical is calculated based on the prediction of the second attention map generated by the lexical prediction model. as well as Select the one with the highest relevance score. k A set of lexical units are provided for inclusion in the first subset of lexical units.

21. The method of claim 20, wherein k It is selected based on the attention ratio, which is defined as the percentage of total input lexical units to be processed by the second attention layer of the machine learning model.

22. The method of claim 16, wherein the first attention map and the second attention map comprise attention maps for a first attention head in the machine learning model.

23. The method of claim 22, wherein identifying the first subset of lexical units and the second subset of lexical units comprises predicting relevant lexical units based on a concatenation of the attention map for the first attention head and attention maps for one or more additional attention heads in the machine learning model.

24. The method of claim 16, wherein the lexical prediction model comprises a convolutional model configured to generate a predicted attention map for a second attention layer of the machine learning model based on an input of a first attention map generated by the first attention layer of the machine learning model.

25. The method of claim 16, wherein the lexical prediction model comprises a model configured to generate normalized attention maps for a plurality of attention heads in the machine learning model.

26. A processor-implemented method for machine learning, the method comprising: Use machine learning models to generate multiple benchmark ground value attention maps for a set of inputs in the training dataset; A word prediction model is trained based on the training dataset to generate a predicted attention map, wherein the word prediction model is trained based on minimizing the difference between the predicted attention map and the corresponding benchmark ground attention map from the plurality of benchmark ground attention maps, and wherein the predicted attention map includes a plurality of words, each corresponding word being associated with a corresponding relevance score generated based on the predicted attention map. as well as Deploy the lexical prediction model.

27. The method of claim 26, wherein the lexical prediction model comprises a convolutional model trained to generate the predicted attention map for a second attention layer of the machine learning model based on input of a first attention map generated by a first attention layer of the machine learning model.

28. The method of claim 26, wherein the lexical prediction model comprises a model trained to generate normalized attention maps for multiple attention heads in the machine learning model.

29. The method of claim 26, wherein training the lexical prediction model comprises training the lexical prediction model based on minimizing the Kolb-Leibler (KL) divergence loss between the predicted attention map and the corresponding ground truth attention map.

30. The method of claim 26, wherein the machine learning model comprises a frozen transformer neural network.

Images