Serial composite high efficiency converter with multi-scale multi-range attention

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for performing machine learning. In one example, an input data sequence is accessed and sliced based on a slice length hyper-parameter to generate a stacked slice input data representation. The stacked slice input data representation is processed with a slice attention layer to generate a stacked slice output data representation. The stacked slice output data representation is de-sliced to generate a sequence of output data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional patent application No. 18 / 319,259 filed on May 17, 2023, which claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 364,947 filed on May 18, 2022, the entire contents of which are incorporated herein by reference in their entirety.

[0003] introduction

[0004] Various aspects of the present disclosure relate to an efficient transformer-based machine learning model architecture.

[0005] The Transformer network architecture offers state-of-the-art performance and versatility in many fields and has recently been recognized as one of the most important recent advances in artificial intelligence. However, it is well known that Transformer-based model architectures suffer from their O(N) 2 ) complexity and is very expensive in terms of computation and memory requirements, with its complexity growing quadratically with respect to the input length N. This complexity issue often prohibits the use of Transformer-based model architectures for tasks with long sequences of data, and also limits the range of devices on which such model architectures can be deployed.

[0006] Conventional attempts to reduce the complexity of transformer-based model architectures often come at the expense of significant accuracy. Thus, there is a need for improved transformer-based machine learning model architectures. Summary of the invention

[0007] Certain aspects provide a computer-implemented method comprising: accessing an input data sequence; slicing the input data sequence based on a slice length hyperparameter to generate a stacked slice input data representation; processing the stacked slice input data representation using a slice attention layer to generate a stacked slice output data representation; and slicing the stacked slice output data representation to generate an output data sequence.

[0008] Other aspects provide: a processing system configured to perform the aforementioned methods and those described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0009] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings illustrate certain features of the one or more aspects and therefore are not to be considered limiting of the scope of the disclosure.

[0011] Figure 1 Depicting an example of an attention function.

[0012] Figure 2 An example of a model based on a high efficiency converter is depicted.

[0013] Figure 3 Depicted is an example slice attention layer architecture.

[0014] Figure 4 Depicted is an example data flow for slice attention.

[0015] Figure 5 Depicted is an example data flow for slice attention with slice overlap.

[0016] Figure 6 Depicted is an example data flow for slice attention with focus overlap.

[0017] Figure 7 Depicted is an example workflow for focal local attention.

[0018] Figure 8 Depicted is an example method for performing machine learning using slice attention.

[0019] Fig. 9 An example processing system is depicted.

[0020] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0021] Aspects of the present disclosure provide apparatus, methods, processing systems, and non-transitory computer-readable media for an efficient transformer-based machine learning model architecture.

[0022] Transformer-based neural network architectures represent a core technology for modern machine learning and artificial intelligence applications due to their state-of-the-art performance and versatility in many fields. Transformers are one of the most popular modern neural network architectures as they have achieved outstanding results on various types of challenging language tasks and have recently been applied to vision tasks as well.

[0023] However, conventional transformer-based models are very expensive due to their inherent high complexity. Conventional transformers suffer from various problems, including quadratic computation and memory complexity with respect to the length of the input data sequence (e.g., O(N^2) / ... 2 )), and reduced task performance (e.g., reduced accuracy) when modeling longer sequences.

[0024] Previous attempts to address the technical complexity problem with transformer-based models have come at the expense of significant performance tradeoffs. That is, conventional transformer-based models that have become more efficient in terms of complexity have also become less performant (e.g., have reduced accuracy). For example, some transformer designs that are optimized specifically for modeling longer sequences (but add additional overhead for modeling shorter sequences) are generally not generally applicable to different tasks.

[0025] To overcome these and other technical problems of conventional transformer-based model architectures, some aspects described herein relate to efficient transformer-based neural network architectures. In some aspects, the transformer-based neural network architecture uses serial compounding of attentions at different scales applied to stacked slice representations of an input sequence, and / or multi-scale position embeddings applied immediately at attention time. In some aspects, the model architecture described herein may be referred to as a "compound slice transformer." Note that with a fixed slice length L as a hyperparameter, the efficient transformer-based neural network architecture described herein has a complexity of O(NL+N 2 / L 2 ), which is comparable to or even more efficient than the linear complexity in practical settings, and in any case significantly more efficient than the complexity of conventional transformer-based models, O(N 2 ).

[0026] Because the efficient transformer-based neural network architectures described herein involve or use slices of the input sequence, some aspects described herein involve overlapping or focal attention techniques that seamlessly capture token interactions (where a "token" is an element or value in the input sequence) across slice boundaries, thereby preventing context fragmentation. As a result, the efficient transformer-based neural network architectures described herein can achieve competitive performance (e.g., high accuracy) in many different tasks while achieving state-of-the-art performance on the Long Range Arena benchmark, which consists of 5 long sequence classification tasks that evaluate model performance on long sequences. This metric measures both efficiency and performance because the model must handle the N-dimensional space caused by long sequences. 2 Complexity.

[0027] Introduction to Self-Attention

[0028] In various aspects of the present disclosure, a transformer-based architecture is described that utilizes a (self)attention function to draw global dependencies between inputs and outputs. An attention function can generally be described as a function configured to map a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. In some aspects, the output is calculated as a weighted sum of values, where the weight assigned to each value is calculated by a compatibility function of the query and the corresponding key.

[0029] Figure 1 An example attention mechanism 100 is depicted in which an input matrix 102 is weighted by trainable parameters including query weights 103, key weights 105, and value weights 109 to generate a query matrix 104, a key matrix 106, and a value matrix 110, respectively. That is, with one or more learned query weights 103 (denoted as W in the illustrated example), Q ) to generate a query matrix 104 (also referred to in some aspects as a "query"). Sequentially or in parallel, one or more learned key weights 105 (denoted as W in the illustrated example) are used to weight the input matrix 102 to generate a query matrix 104 (also referred to in some aspects as a "query"). K ) to generate a key matrix 106 (also referred to in some aspects as a “key”), and with one or more learned value weights 109 (denoted as W in the illustrated example). V ) are weighted to generate a value matrix 110 (also referred to in some aspects as "values"). In some aspects, these multiplications (to create query matrix 104, key matrix 106, and / or value matrix 110) may be referred to as element-wise or Hadamard multiplications or products.

[0030] In the illustrated example, the query matrix 104 and the key matrix 106 are then aggregated or combined (e.g., using a matrix multiplication of the two matrices 104 and 106), as depicted by arrow 107, to generate an intermediate matrix 108. Note that in the illustrated example, the input matrices may have a size of N×D (e.g., a size of N*D). After applying the learned weights 103, 105, and 109, the resulting matrices may have an equal size of N*D. That is, as illustrated, the query matrix 104 and the value matrix 110 each have a size of N×D (e.g., a size of N*D), while the key matrix 106 has a size of D×N (e.g., a size of D*N).

[0031] However, since intermediate matrix 108 is generated using matrix multiplication of query matrix 104 and key matrix 106 (eg, via arrow 107), intermediate matrix 108 generally has dimensions N×N (eg, size N 2 As discussed above, this results in O(N 2 ) complexity.

[0032] In the illustrated example, the intermediate matrix 108 is then weighted (e.g., multiplied) with the value matrix 110 (using operation 111, which may correspond to a matrix multiplication operation) to generate an output matrix 112, which serves as the output from the attention mechanism 100. In the illustrated example, the output matrix 112 has the same dimensions and size as the input matrix 102 (e.g., size N×D, size N*D).

[0033] Transformers and Multi-Head Self-Attention

[0034] In some aspects, the transformer layer in the neural network model may include a multi-head self-attention sublayer followed by a feed-forward network with an optional cross-attention sublayer (e.g., in the case of a decoder). The multi-head self-attention (e.g., output matrix 112), which may serve as the main source of sequence modeling capabilities of the transformer, is defined as the concatenation of the self-attention outputs in all attention heads:

[0035] Y = concat[Y 0 ,Y 2 ,...,Y H-1 ] (1)

[0036] Each of the output From the input (e.g., input matrix 102) computes the scaled dot product attention:

[0037]

[0038] With query Q h =XW q,h (e.g., query matrix 104 generated by multiplying input matrix 102 with query weight 103 for a particular head h), key K h =XW k,h (e.g., a key matrix 106 generated by multiplying the input matrix 102 with the key weight 105 for a particular head h) and the value V h =XW v,h (e.g., a value matrix 110 generated by multiplying the input matrix 102 by the value weights 109 for a particular head h) as a linear transformation of the input X. In some aspects, the weights (e.g., query weights 103, key weights 105, and / or value weights 109) may be implemented as scalar values ​​and / or matrices (e.g., where the query weights 103, key weights 105, and value weights 109 may each include a weight matrix). Here, it is assumed that the query, key, and value have the same hidden size d h =D / H. Therefore, in the following, the header index h and the scaling factor are omitted for simplicity. The query is represented as the query location index i and the key and value are denoted as kj and v j Similar to j , the attention output at the location of the i-th token corresponds to: at:

[0039]

[0040] Due to the non-linear and normalization characteristics of the softmax function, calculate is performed to obtain the attention weights, and then these values are aggregated. Therefore, for A, the dot product and the calculation complexity of aggregating the values of the attention weights for AV are both O(N 2 )(the memory complexity is also O(N 2 ))). Therefore, self-attention has quadratic complexity with respect to the sequence length N.

[0041] Abstract Attention

[0042] Assuming that softmax dot-product attention plays an important role in the sequence modeling ability of the transformer model, the abstract attention maintains the form of the basic attention calculation according to Equation 3.

[0043] In aspects of the present disclosure, abstract attention can be defined as a series of efficient attention methods, where the length of the attention operands is reduced to M (< N) by applying an abstraction function, thereby correspondingly reducing the complexity of the attention. Abstract attention can be further classified into resolution-preserving or non-preserving attention, where the operands to be abstracted are selected according to these attentions, and the preservation of resolution is between the input and output sequences. That is, resolution-preserving attention preserves the resolution of the input sequence, while non-preserving resolution does not. In some aspects, when the query (e.g., query matrix 104) is abstracted, this attention is called resolution-non-preserving attention, and the abstracted attention also produces an abstracted output. In some aspects, this classification as preserving or non-preserving attention is determined according to a given task. For example, tasks such as language modeling and machine translation usually rely on high (or full) resolution at the output to be maintained. In those cases, in some aspects, only the keys (e.g., key matrix 106) and values (e.g., value matrix 110) are abstracted while maintaining the query resolution. The abstract resolution-preserving attention in this case can be expressed as follows:

[0044]

[0045] where Ω j′ represents the abstraction range with cardinality |Ω j′ | = M j′ for abstracting k′ k of the j'-th key, and is a many-to-one abstraction function. Abstracting the value Vj′ It can be expressed similarly to Equation 6.

[0046] Resolution-non-preserving abstractions can be used for tasks where output resolution is not necessary or not very important, such as sequence-level classification problems. However, by additional processing that exploits representations at lower layers (e.g., using cross-attention with input tokens), it is possible to restore resolution in some aspects. Together with the key and value abstractions (discussed above with reference to Equations 5 and 6), in some aspects, the query can be abstracted as:

[0047]

[0048] And the resolution non-preserving attention can be defined as:

[0049]

[0050] The attention output vector y i′ At each abstract location i′, we obtain: y It can be defined as:

[0051]

[0052] In some aspects of the Transformer-based architectures described herein, since the output of the local attention maintains high (or full) resolution (e.g., because the query is not abstracted), a simple broadcast function can be used to recover the sequence length, i.e., y i =y i′ , for i∈Ω i′ , rather than restoring the resolution. Note that the term broadcasting, as used herein, describes how arrays with different shapes are handled during arithmetic operations. Subject to certain constraints, a smaller array can be "broadcasted" across a larger array so that they have compatible shapes (e.g., by copying or duplicating elements of the array to create an array of the desired size). Broadcasting provides a means of vectorizing array operations.

[0053] Multi-scale and multi-range attention

[0054] While some previous abstract attention and non-attention approaches have achieved sub-quadratic complexity (and even linear complexity for some approaches), these existing approaches often come at the expense of degraded performance (e.g., reduced accuracy) on a baseline. However, the efficient transformer-based model architecture described in this paper exploits multi-scale attention by combining local and global attention and provides significant accuracy improvements (often outperforming conventional architectures) while still maintaining efficiency benefits. Figure 2 The example is described in more detail based on the model of the high efficiency converter and the following reference Figure 3Discussing the example slice attention architecture in more detail.

[0055] In some aspects, local attention (also known as sliding window attention) limits the attention range to the vicinity of the query position. That is, the entire abstraction range can be utilized to perform key abstraction, and the position-dependent abstraction function can be used to perform query abstraction:

[0056]

[0057] Where H is the Heaviside step function, w is the window length, and ⊙ is the element-wise product. Thus, in some aspects, local attention can be defined using the following Equation 10:

[0058]

[0059] In some aspects, for better computational efficiency, for block-wise attention, the block-wise key abstraction can be defined as For block index b, t i =(b-1 / 2)w, so that (b-1)w≤i <bw。

[0060] In some aspects, for global attention, abstracted attention may be used with position abstraction (which may be loosely viewed as having patch embeddings in a visual transformer (ViT)) and / or context abstraction.

[0061] In some aspects, compound attention (with multi-scale and multi-range components) can be classified according to how the two attentions are combined. For example, one combination method is to concatenate the abstraction of the multi-scale key and the value for a single attention, such as using the following equation 11.

[0062]

[0063] In some aspects, multi-scale attention compounding may be defined using separate attention at different scales, where the output of each is combined or summed (possibly with some weighting coefficients), such as defined using Equation 12 below.

[0064] y i =y l,i +ψ y (y g,i ) (12)

[0065] In the latter case (where the outputs are summed or otherwise combined), other non-attention methods (such as kernel methods) may be used additionally or alternatively to global attention.

[0066] In some aspects, the efficient transformer-based model architecture described herein may correspond to the latter case, where local and global attention are performed separately and their outputs are combined (e.g., summed) together. However, unlike other architectures (such as transformer-in-transformer (TNT)) that have independent (parallel) paths for local attention and global attention and thus prevent information exchange between patches, the efficient transformer-based model architecture described herein uses serial connections between multi-granular attention to achieve bidirectional information routing. Therefore, aspects of the present disclosure may be more suitable for modeling highly non-stationary data (such as natural language text data where the locality assumption does not hold).

[0067] Attention with Input Slice Representation

[0068] Various aspects described herein implement so-called "slice attention" in a transformer-based model (hence the term composite slice transformer), which replaces the full softmax dot-product attention of a conventional transformer model. Beneficially, slice attention utilizes high-resolution attention and abstract attention within a limited range to capture full-range interactions. Unlike previous approaches, in some aspects, multi-scale multi-range attention is configured using a serial connection that allows bidirectional information routing between two attention mechanisms.

[0069] In a high-level description, the multi-scale multi-range attention of the composite slice transformer model corresponds to a combination of block-wise local window attention and patch-based attention. In some aspects, at the embedding layer, the composite slice transformer model transforms the input sequence X into a vector by slicing it based on a fixed length L. Convert to slice stack (e.g., an input sequence of tokens is depicted as a set of slices, each slice having a length of L tokens. In some aspects, a slice length hyperparameter (e.g., a hyperparameter used to define the slice length) L can be selected or defined using a variety of criteria or techniques, and can generally include any value. For example, the slice length can be selected (e.g., by a data scientist) to balance complexity and / or improve model accuracy (e.g., using trial and error to test multiple slice lengths). In some aspects, two attentions with different granularities can then be performed sequentially in each direction, as described below with reference to Figure 3 Discussed in more detail.

[0070] In some aspects, local attention is first performed across tokens within each slice (e.g., see below for Figure 3 315 in more detail), while treating the number of slices as a batch. In some aspects, the slice size N / L can be combined with the batch size and parallelized together so that

[0071]

[0072] Where Q l , K l and V l (respectively) is the learning weight W q,l , W k,l and W v,l Apply the query, key, and value of the local attention obtained by stacking or slicing S. Next, in some aspects, the size of length L can be used in the local attention output using the abstraction function φ y Collapse to get slice embedding In some examples, simple average pooling can be used, where l is the token index along the length dimension, and m l is the attention mask value. In some aspects, normalizing by the sum of the masks in each slice rather than the slice length helps to avoid bias in the average computation caused by the mask tokens.

[0073] In some aspects, a second attention (e.g., global attention) is then performed across slice sizes (e.g., see below with reference to Figure 3 ) to model the full range of information routing at a reduced resolution according to the following conditions:

[0074]

[0075] Where Q g , K g and V g (respectively) by applying W q,g , W k,g and W v,g The query, key, and value to be stacked or sliced ​​S to obtain global attention.

[0076] Volatile transient multi-scale position embedding

[0077] Because transformer-based models typically do not contain recurrence and convolution, in some aspects, some information about the relative or absolute positioning of tokens in the sequence is injected so that the model can exploit the order of the sequence. This can be referred to in some aspects as positional embedding (e.g., in some aspects as P for local position embedding). l and P for the global position embedding g , and in Figure 2 In the examples, the embedded functions 207 and 209 are indicated respectively, and in Figure 3 In some aspects, the position encodings are generally of the same size as the token embeddings (e.g., in Figure 2 The embedding layer 202 and / or Figure 3 ), so that the two can be directly summed.

[0078] In some aspects, because both global and local attention are reduced in length (and may have different granularity) in the composite slice transformer model described herein, full positional embeddings of the maximum input sequence length are no longer necessary (compared to conventional architectures). Thus, in some aspects, for local attention, the positional embedding length may be limited to the attention span (e.g., limited to the slice length L). Additionally, because tokens from each slice are aggregated for global attention, a single positional embedding of length N / L at the scale of the slice embedding may be more natural than aggregating full-resolution full-length positional embeddings.

[0079] Therefore, in some aspects of the composite slice transformer model described in this paper, the multi-scale position embedding and Can be used (as shown below Figure 3 ). As discussed in more detail below, these multi-scale position embeddings can be used in a number of ways that are different from conventional transformer models. First, instead of adding position embeddings to stacked slices of token embeddings at the embedding layer, position embeddings can be applied at corresponding attentions in each layer before linear transformation. Second, position embeddings in the disclosed composite slice transformer model can be added only to queries and keys (not values). This prevents the problem of position embeddings accumulating across all layers (and thus undesirably dominating contextual information at the top layer), which potentially leads to performance degradation. Therefore, in some aspects, for the composite slice transformer model, Equations 13 and 14 can be rewritten as:

[0080]

[0081]

[0082] where Y l is the output from local attention, and Y g is the output from global attention.

[0083] Complexity and storage improvements

[0084] In some respects, the quadratic complexity of the conventional transformer model is O(N 2 ), the composite slice transformer model described in this paper has a linear plus decimation quadratic complexity of O(NL)+O(N 2 / L 2 ). However, because the slice length L is typically smaller than the abstract length M, the composite slice transformer model has comparable efficiency to other efficient transformer models for input sequences of practical length among other models with linear complexity.

[0085] Another benefit of using the stacked slice representation in various aspects described herein is the reduction in storage for position embeddings. Since the length of attention is L and N / L for local and global attention, respectively, the composite slice transformer model has fewer parameters (e.g., N*D parameters in a conventional transformer model) than the parameters of a conventional position embedding. parameter).

[0086] Example Composite Slice Transformer Model Architecture

[0087] Figure 2 An example of an efficient transformer based model architecture 200, referred to herein as a compound slice transformer model, is depicted.

[0088] As illustrated, input data 201 (e.g., a sequence of tokens or elements) is provided to an embedding layer 202, which converts the size The input data 201 is transformed into a numerical representation, such as the size where the length of the sequence is N and the size of each element in the sequence is D.

[0089] In the illustrated example, the numerical representation (output from the embedding layer 202) is then provided as input to the slice attention module 205.

[0090] In this example, the slice attention module 205 (also referred to in some aspects as an attention head) begins with a normalization layer 206 that normalizes the input data representation (e.g., using layer normalization) and then provides the normalized input data representation to a slice attention layer 208 (e.g., a layer of a neural network that implements or performs slice attention). Figure 3 An example describing the slice attention layer architecture in more detail. That is, Figure 3 The slice attention layer architecture of block 306 of FIG. 304 may provide additional details of the components and / or operations of the slice attention layer 208. In addition to the normalized input data representation as illustrated, the slice attention layer 208 also receives the local position embedding P l and the global position embedding P g As input, they are generated by embedding functions 207 and 209, respectively, based on the output data representation from embedding layer 202. The output of slice attention layer 208 is typically the output data representation, where local and global attention have been applied (as described below with reference to Figure 3 described in further detail).

[0091] As illustrated, the input to the slice attention layer 208 (via the skip connection 211) and the output of the slice attention layer 208 are then summed at the adder 213 to generate the input for another normalization layer 210. In some aspects, the skip connection 211 is useful for stabilizing gradients and helping training convergence.

[0092] The output from the normalization layer 210 (normalized output data representation) is then provided to a feed-forward network (FFN) 212, which can be configured as a point-by-point fully connected feed-forward network so that the attention output is nonlinearly transformed into a new representation for the next layer. Here again, a skip connection 215 can be used to add the input of the normalization layer 210 to the output of the feed-forward network 212 through an adder 217 to generate the final output data 214 from the transformer-based model architecture 200.

[0093] Although the illustrated example depicts a single slice attention module 205 (or attention head) for simplicity and conceptual clarity, in various aspects, there may be multiple slice attention modules 205 implemented in the architecture 200 (e.g., the architecture 200 may use a multi-head slice attention mechanism).

[0094] also, Figure 2 Only one example of a composite slice transformer model architecture is depicted, and variations may be made while maintaining the underlying slice attention functionality. For example, the ordering of normalization layers may be changed from a "pre-normalization" configuration as depicted in architecture 200 (e.g., where normalization layers 206 and 210 use / provide inputs to slice attention layers 208 and FFN 212, respectively, immediately before the slice attention layers and FFNs) to a "post-normalization" configuration (e.g., where normalization layers 206 and 210 use / receive their inputs from adders 213 and 217, respectively). Such post-normalization configurations are not shown in the depicted example. Similarly, in some aspects, architecture 200 may abandon or exclude skip connections 211 and / or 215.

[0095] Example Slice Attention Layer Architecture

[0096] Figure 3 An example slice attention layer architecture 300 is depicted. In some aspects, the architecture 300 is directed to Figure 2 Specifically, in some aspects, block 306 may correspond to the slice attention layer 208, and the components and operations depicted therein may include Figure 2 In the slice attention layer 208.

[0097] As illustrated, input 305 (with size ) is provided to a slicing layer 310, which slices the sequence based on a slice length hyperparameter L to generate N / L slices of the input 305, each slice having a length L. In some aspects, L is a factor of N, allowing the input to be sliced ​​into an integer number of slices. In some aspects, L may not be a factor of N, and padding may be added to one or more of the slices in the slice to form an integer number of slices of equal length. These slices are then stacked (as described below with reference to Figure 4 Discussed in more detail) to generate That is, the stacked slice input data representation can be formed by concatenating or stacking the slices to form an aggregate tensor.

[0098] As mentioned above Figure 2 As discussed, the input is thus used to form a stacked slice data representation (S in the above description). Then, at section 315, by first adding the embedding layer 312 (which may correspond to Figure 2 The local position of the embedding layer 202) is embedded in P l (output by embedding function 314, which may correspond to Figure 2 The embedding function 207 of 207 is added to the input data for generating keys and queries, rather than the input data for generating values ​​(as described above), and a first local (high-resolution or full-resolution) attention is performed on the input data. Then, the local attention parameter set 325A to 325C (denoted as W in the illustrated example) is added to the input data for generating keys and queries, rather than the input data for generating values ​​(as described above). q,l , W k,l and W v,l ) is applied to the stacked slice data representation (augmented by local position embeddings in the case of keys and queries) to generate a local query Q l , local key K l and the local value V l In some aspects, the local attention parameters 325 may be referred to as a local set of weights, a local set of trained weights, a local set of learned weights, a first set of weights, a first set of trained weights, a first set of local weights, etc. As described above, a matrix multiplication is then performed at the local attention element 330 to generate a size The local attention output data.

[0099] That is, the local attention mechanism (indicated by segment 315) includes adding the local position embedding at adder 320, applying the local attention parameters 325 (also referred to as weights), and finally using the local attention element 330 (e.g., such as by using equation 15 above to calculate the local attention). In general, the illustrated example depicts performing the local attention (in segment 315) in a particular arrangement (e.g., including the use of position embeddings for a subset of matrices). However, other configurations may be used in some aspects (e.g., position embeddings may be added to the value matrix as well as the key and query matrices, position embeddings may be excluded or not used for one or more of the matrices, etc.).

[0100] In some aspects, as discussed above, the local attention parameters 325 are trainable (e.g., learned) parameters. In some aspects described herein, the first (local) attention is referred to as high resolution. As used herein, the local attention may be referred to as "high" resolution to indicate that the local attention uses or has a higher resolution (e.g., up to and including full resolution) than the second (global) attention. That is, in some aspects, global attention may be performed at a reduced resolution (e.g., by abstracting or aggregating one or more tokens or elements in a sequence into a sequence with fewer elements, such as by grouping multiple elements into a single element, and performing global attention on this relatively small sequence compared to the length of the original sequence). This can improve efficiency and computational cost. In some aspects, local attention may be performed at a relatively high resolution (e.g., with less abstraction, such as by aggregating fewer elements together, and / or by not using abstraction, such as by evaluating slices at full (original) resolution).

[0101] In the illustrated example, the local attention output data (output by the local attention element 330) is then processed by the slice embedding element 335 to resize the data to As described above, the slice embedding element 335 can implement an abstraction function, such as average pooling within each slice in some examples, to generate a slice embedding. As discussed below, since global attention uses a relatively low resolution (compared to operating on the original input tokens), this abstraction (e.g., average pooling within each slice) allows global attention to operate more efficiently or at a reduced cost.

[0102] As illustrated, at section 345, by first adding at adder 350 the embedding layer 312 (which may correspond to Figure 2 The global position embedding P of the embedding layer 202) g (by embedding function 344 (which may correspond to Figure 2The output of the embedding function 209) is added to the local attention output data for generating keys and queries, rather than the input for generating values, and a second global (and reduced or low-resolution) attention is performed on the slice embedding. Note that unlike the local position embedding P l Different, the global position embedding P g By size Same size as the slice embedding.

[0103] As illustrated, the global attention parameter set 355A to 355C (represented in the illustrated example as W q,g , W k,g and W v,g ) is applied to the slice embeddings (for both key and query, augmented by global position embeddings) to generate the global query Q g , Global Key K g and the global value V g In some aspects, the global attention parameters 355 may be referred to as a global set of weights, a global set of trained weights, a global set of learned weights, a second set of weights, a second set of trained weights, a second set of local weights, etc. As described above, a matrix multiplication is then performed at the global attention element 360 to generate a matrix having size The global attention output data.

[0104] That is, the global attention mechanism (indicated by segment 345) includes adding the global position embedding at adder 350, applying global attention parameters 355 (also called weights), and finally using the global attention element 360 (e.g., such as by calculating the global attention using equation 16 above).

[0105] In some aspects, as discussed above, the global attention parameters 355 are trainable (e.g., learned) parameters. In some aspects described herein, the second (global) attention is referred to as low resolution and / or reduced resolution. As used herein, this global attention may be referred to as "low" or "reduced" resolution in some aspects to indicate that the global attention uses or has a lower resolution than the resolution of the first (local) attention (e.g., the input to the global attention may be abstracted or otherwise reduced to a smaller number of tokens or elements than the original input sequence). In some aspects, rather than reducing the resolution, the global attention may similarly operate at full (or higher) resolution in a manner similar to the local attention.

[0106] In the illustrated example, the output from the global attention element 360 is then broadcast added to the local attention output (output by the local attention element 330) via the skip connection 340 and the adder 365. Here, the adder 365 is added due to the output from the global attention element 360. With local attention output The broadcast addition is performed based on the difference in magnitude between the two.

[0107] As depicted, the output of adder 365 is then provided to a remove slice layer 370, which transforms the output from a stacked slice shape to a sequence shape The original input data is thereby matched with the slice layer 310 .

[0108] Finally, the linear layer 375 performs a linear transformation to generate stacked slice output data 380.

[0109] Figure 4 Describe as can be made about Figure 3 An example data flow 400 for slice attention implemented by the described slice attention layer architecture 300.

[0110] As depicted, an input data sequence 405 (e.g., Figure 3 The input 305) is sliced ​​via operation 410 (e.g., based on the use of Figure 3 The slice length hyperparameter of the slice layer 310) to generate a stacked slice representation 415. The stacked slice representation is then processed by all or part of the slice attention layer (e.g., the local attention element 420 (e.g., Figure 3 Segment 315)) of which may have complexity to generate a local attention output 435. As discussed above, in some aspects, the local attention element can be referred to as "high resolution". In the illustrated example and as discussed above, the local attention element 420 generally includes applying trained or learned weights (e.g., key weights and / or query weights having values ​​learned during training of the model) to each slice of the stacked slice representation 415 (thereby generating a query matrix 425B (e.g., Figure 1 425A) and the key matrix 425A (e.g., Figure 1 These matrices 425 are then combined (e.g., using matrix multiplication) to generate an intermediate matrix 430 (e.g., Figure 1 The intermediate matrix 108 of , which is then combined with the value matrix (e.g., Figure 1 The value matrix 110 of the stacked slice representation 415 is similarly generated using trained or learned weights, such as value weights with values ​​learned during training of the model) are combined (e.g., using matrix multiplication) to generate an output local attention for the slice. Although the illustrated example depicts applying local attention to a single slice, in various aspects, the local attention element 420 can operate on the entire stacked slice representation 415. In addition, while the above discusses generating and using one or more weights to generate key, query, and value matrices, in some aspects, local attention can generally include a wide variety of operations to generate local attention outputs.

[0111] As illustrated, the abstract function 440 (e.g., Figure 3 The local attention output 435 is processed by the slice embedding element 335 of the global attention element 455 to generate a slice embedding 450. The slice embedding 450 is then processed by the global attention element 455 (e.g., Figure 3 345), which may have a complexity To generate a global attention output 470. As discussed above, due to this abstraction function 440, the global attention element can be referred to as "reduced resolution" in some aspects. That is, because global attention can be performed on slice embeddings 455 (generated by abstraction abstraction function 440) rather than directly on input tokens, global attention can be considered to be a relatively low resolution compared to local attention. As discussed above, the global attention element 455 can generally apply learned parameters (e.g., key weights and / or query weights) to generate a query matrix 460B and / or a key matrix 460A, which are combined to create an intermediate matrix 465, which is then combined with the value matrix to produce the global attention output 470.

[0112] As illustrated, the global attention output 470 is then passed through an adder 475 (e.g., Figure 3 The adder 365 of 4 is broadcast added to the local attention output 435 (provided via the skip connection 445) to generate the stacked slice output data 480. Finally, the stacked slice output data is de-sliced ​​using operation 485 (e.g., using Figure 3 ) to provide an output data sequence 490 (eg, slice output data 380).

[0113] Overcoming context fragmentation - overlapping local and focal attention

[0114] In order to avoid context fragmentation with the slice data representation used in the composite slice transformer model, overlapping attention can be used in some aspects. That is, in some aspects, context fragmentation can be caused by the local attention being strictly limited to considering only other elements within the same slice, meaning that elements near the beginning and end of each slice can lose valuable context contained in one or more elements in neighboring slices. By using overlapping attention, in some aspects, this context fragmentation can be reduced or avoided.

[0115] Figure 5 Describe as can be made about Figure 3 The described slice layer architecture implements an example data flow 500 of slice attention using overlapping slice attention (in some aspects referred to as overlapping slice local attention). Flow 500 is in the same Figure 4400 , however, the local attention element 520 uses overlapping local attention where the slices are overlapped to regain contextual information lost by the slicing operation. That is, allowing slices to overlap may allow local attention to be generated for each element with a more complete context of the element (e.g., based on additional neighboring elements), rather than using strictly non-overlapping slices that fragment the context of some elements in the slice. It can be seen that the overlap does come at the expense of additional complexity based on the overlap ratio, thereby increasing the overall complexity to where a is a hyperparameter that specifies the amount of overlap.

[0116] In some aspects, overlapping local attention is implemented by generating local attention outputs 535 based on overlapping slices in the stacked slice representation 515. For example, in the illustrated aspect, the local attention element 520 computes the local attention output 535 based on the concatenated slice pairs (e.g., by doubling the width of the key vector 525A (also referred to in some aspects as a local key vector, matrix, or tensor) and the value vector (also referred to in some aspects as a local value vector, matrix, or tensor)).

[0117] In some aspects, to address the complexity effects from overlapping attention when using sliced ​​data representations, focal attention (also referred to in some aspects as focal slice attention) can be utilized as a more efficient way to create overlaps. Figure 6 Describe as can be made about Figure 3 An example data flow 600 of slice attention using focal local attention is described in detail in the slice layer architecture. Flow 600 is similar to Figure 5 500 is performed in much the same manner, however, segment-wise focused local attention is used at the local attention element 620 (depicted by elements 625A and 625B), which is about Figure 7 , described in more detail. Unlike some conventional attempts at focal attention, in various aspects described herein, a sequence of one-dimensional slices is used to achieve intermediate attention between local attention and global attention. This can be performed by taking sequences of different overlapping lengths of the query sequence, as described above with reference to Figure 5 Assume that we take slice N of the input sequence w(l-1):wl , where w is the width of the slice, and l is the lth slice in the sequence. This forms the query matrix. For the key and value matrices, a number of different sequence lengths can be used. For example, the system can use the following four sequence lengths:

[0118] (K,V) w(l-1):w l

[0119] (K,V) w(l-1-α):wl+a

[0120] (K,V) w(l-1-2α):wl+2a

[0121] (K,V) w(l-1-4α):wl+4a

[0122] In the above expression, α is an optional overlap ratio. In some aspects, the key and value sequences may then be passed through different pooling and / or convolution operations to merge information, as described below with reference to Figure 7 In more detail. In general, the longer the sequence, the larger the pooling and the coarser the information in the output sequence. These sequences can then be concatenated and convolved to achieve the original size as the query, which allows reducing the complexity of the composite slice transformer from arrive

[0123] Figure 7 An example workflow 700 for implementing focused slice local attention is depicted. As discussed above, the stacked slice representations may be transformed using different sequence lengths to create overlaps. These difference sequences may then be individually pooled or convolved, and then concatenated to generate the overlapped stacked slice representations. The overlapped stacked slice representations may then be convolved to reshape the data so that local attention may be applied and a focused local attention output may be generated (which may then be processed by the rest of the composite slice transformer architecture as described above).

[0124] Specifically, in the illustrated example, an input data sequence 705 (eg, Figure 4 405), and using a data slicing operation 710 (e.g., Figure 3 The stacked slice representation 715 is generated using overlapping slices, as described above with reference to FIG. Figure 5 and Figure 6 In some aspects, as discussed below, the slicing operation 710 may generate slices of multiple different lengths (defined by a slice length hyperparameter) and / or having multiple different amounts of overlap (defined by an overlap hyperparameter). That is, the slicing operation may be used to generate multiple stacked slice representations, each with a different length or size. In the illustrated example, via operation 720, the stacked slice representation 715 is used to generate a query matrix 725. For example, operation 720 may correspond to adding a local query weight (e.g., Figure 1 A query weight of 103 and / or Figure 3 The local attention parameters 325C) are applied to the stacked slice representation 715 to generate a query matrix 725 (e.g., Figure 1 Query matrix 104).

[0125] In the illustrated example, via operations 730A-730C, the system may also generate an intermediate set of tensors or matrices 735A-735C (collectively referred to herein as "tensors 735" or "matrices 735") that are used to generate key and value matrices 745A-745C for attention operations, such as by using operations 740A-740C (e.g., convolutions), as discussed below. In the illustrated example, the intermediate matrices 735 may correspond to the value matrices (e.g., using Figure 1 Value weight 109 and / or Figure 3 ) and / or a key matrix (e.g., using Figure 1 A key weight of 105 and / or Figure 3 That is, the same operation can be used to generate each overlapping stacked slice representation, followed by a convolution operation to reshape each overlapping stacked slice representation to the same size as the query matrix 725.

[0126] As illustrated, operation 730 corresponds to applying key weights and / or value weights to stacked slice representation 715 to generate intermediate matrices 735A to 735C. As illustrated, each operation 730 corresponds to a matrix of a different size. Specifically, if query matrix 725 is Q w(l-1):wl (e.g., a first size, such as w(l-1) times wl), then the intermediate matrix 735A has the same size (e.g., K w(l-1):wl For the bond matrix, and V w(l-1):wl As shown, the intermediate matrix 735B is larger (e.g., K w(l-1)-1:wl+1 For the bond matrix, and V w(l-1)-1:wl+1 For value matrix) rather than intermediate matrix 735A. Similarly, intermediate matrix 735C is larger than intermediate matrix 735B (e.g., K w(l-1)-2:wl+2 For the bond matrix, and V w(l-1)-2:wl+2 For the value matrix). In this way, because the intermediate matrices 735B and 735C include additional elements that overlap with adjacent slices in the stacked slice representation 715, the system can prevent context fragmentation by generating local attention based in part on these overlapping elements.

[0127] In the illustrated example, the intermediate tensors 735 are then processed via convolution operations 740 to generate new sets of intermediate tensors 745. As illustrated, the system generally uses larger convolution kernels for larger intermediate tensors 735 (thereby reducing the size of the resulting intermediate kernels 745). Specifically, in the illustrated example, convolution operation 740A does not change the size of intermediate matrix 735A (e.g., using a 1×1×d×d convolution), convolution operation 740B produces an intermediate matrix 745B that is slightly smaller than intermediate matrix 735B (e.g., using a 2×1×d×d convolution), and convolution operation 740C produces an intermediate array 745C that is significantly smaller than intermediate matrix 735C (e.g., using a 3×1×d×d convolution).

[0128] In various aspects, the actual size of the intermediate tensors or matrices 735 and / or convolution operations 740 may vary depending on the particular implementation (e.g., depending on the value of a). Additionally, while three intermediate tensors 735 are depicted, in various aspects, the system may generate any number of intermediate tensors 735 of various sizes.

[0129] As illustrated, the intermediate tensors 745A-745C are then concatenated via operation 750 to generate an overlapping stacked slice representation 755. Since this overlapping stacked slice representation 755 is substantially larger than the query matrix 725, in the illustrated workflow 700, a convolution operation 760 is used to reshape the overlapping stacked slice representation 755 and change its size to match the dimensions of the query matrix 725. For example, in the illustrated aspect, a 1×1×17×8 convolution is used to generate a matrix 765 (e.g., a key matrix in the case where operation 730 uses key weights and / or a value matrix in the case where operation 730 uses value weights). In some aspects, as discussed above, in the case of a key matrix, operation 760 may also include a transpose operation (e.g., to prepare the key matrix for matrix multiplication using an attention mechanism).

[0130] In the illustrated example, the matrix 765 (e.g., the key matrix and the value matrix generated using overlapping slices) and the query matrix 725 are then provided to the local attention mechanism 770 (e.g., Figure 3 The local attention element 330 and / or Figure 6 ), which generates a focused local attention output 775, as discussed above. In the illustrated example, because overlapping slices are used to generate the key and value matrices, the focused local attention output 775 can prevent or reduce context fragmentation, thereby resulting in improved model accuracy (without using reference Figure 5 The additional overhead introduced by the overlapping method described in .

[0131] Example Method

[0132] Figure 8 Depicted is an example method 800 for performing machine learning utilizing slice attention.

[0133] Method 800 begins at block 802 by accessing an input data sequence, such as described above with respect to input 305 and Figure 3 described.

[0134] At block 804, the input data sequence is sliced ​​based on a slice length hyperparameter to generate a stacked slice input data representation, such as described above with respect to Figure 3 and slice layer 310 .

[0135] At block 806, the stacked slice input data representation is processed using a slice attention layer to generate a stacked slice output data representation, such as described above with respect to Figure 2 and slice attention layer 208 and about Figure 3 The slice attention layer architecture 300 (e.g., sections 315 and / or 345) is described.

[0136] At block 808, the stacked slice output data representation is de-sliced ​​to generate an output data sequence, such as described above with respect to Figure 3 and removing the slice layer 370 as described above.

[0137] In some aspects, processing the stacked slice input data representation using a slice attention layer to generate a stacked slice output data representation includes: processing the stacked slice input data representation using a high-resolution local attention layer (e.g., Figure 3 315) to generate local attention output data; processing the local attention output data using a slice embedding layer (e.g., Figure 3 335 ) to generate a slice embedding; processing the slice embedding using a reduced-resolution global attention layer (e.g., Figure 3 345) to generate global attention output data; and performing a broadcast addition of the local attention output data and the global attention output data (e.g., via Figure 3 Adder 365) to generate a stacked slice output data representation. One advantage of this aspect is that high-resolution local attention can be used to accurately generate local attention, while reduced-resolution global attention can be used to generate global attention at a reduced computational cost.

[0138] In some aspects, processing the stacked slice input data representation using a high-resolution local attention layer includes applying a first set of training weights (e.g., Figure 3 325) is applied to the stacked slice input data representation, and processing the slice embeddings with a reduced-resolution global attention layer includes applying a second set of trained weights (e.g., Figure 3The global attention parameters 355) are applied to the slice embedding. One advantage of this aspect is that the local and global attention layers can use different sets of trained weights, which can improve model performance.

[0139] In some aspects, processing the stacked slice input data representation using a high-resolution local attention layer includes: applying a first set of training weights (e.g., Figure 3 The local attention parameters 325 of the stacked slice input data representation are applied to generate a local key vector (e.g., a key matrix for local attention), a local query vector (e.g., a query matrix for local attention), and a local value vector (e.g., a value matrix for local attention); and local attention output data is generated based on the local key vector, the local query vector, and the local value vector. One advantage of this aspect is that the local attention can be generated using weights learned during training for high-resolution local attention.

[0140] In some aspects, processing the stacked slice input data representation with a high-resolution local attention layer further includes embedding the local positions (e.g., via Figure 2 and Figure 3 The embedding function 207 of is added to the local key vector and the local query vector, and the length of the local position embedding is based on the slice length hyperparameter. One advantage of this aspect is that the position embedding can be customized to take into account local positioning based on the slice.

[0141] In some aspects, processing the slice embedding with a reduced-resolution global attention layer includes: Figure 3 The global attention parameters 355 of the slice embedding are applied to generate a global key vector (e.g., a key matrix for global attention), a global query vector (e.g., a query matrix for global attention), and a global value vector (e.g., a value matrix for global attention); and global attention output data is generated based on the global key vector, the global query vector, and the global value vector. One advantage of this aspect is that the global attention can be generated using weights learned during training for reduced resolution global attention.

[0142] In some aspects, processing the slice embedding with the reduced-resolution global attention layer includes embedding the global position (e.g., via Figure 2 and Figure 3 The embedding function 209 of is added to the global key vector and the global query vector, and the length of the global position embedding is based on the input data sequence length divided by the slice length hyperparameter. One advantage of this aspect is that the position embedding can be customized to take into account global positioning.

[0143] In some aspects, processing the stacked slice input data representation using a high-resolution local attention layer includes performing overlapping slice local attention, such as described above with reference to Figures 4 to 5 In some aspects, slicing the input data sequence is further performed based on an overlap hyperparameter to generate overlapping slices of the input data sequence. An advantage of this aspect is that overlapping slices local attention can reduce or prevent context fragmentation, and / or overlapping slices can improve model accuracy.

[0144] In some aspects, processing the stacked slice input data representation using a high-resolution local attention layer includes performing focal slice local attention, such as described above with reference to Figure 6 to Figure 7 In some aspects, slicing the input data sequence includes generating multiple slices having multiple sequence lengths, and performing the focused slice local attention includes: generating multiple intermediate tensors based on the multiple slices; and aggregating the multiple intermediate tensors. An advantage of this aspect is that the focused slice local attention can reduce or eliminate context fragmentation, and / or aggregating the intermediate tensors can reduce computational costs.

[0145] In some aspects, the slice attention layer includes multiple slice attention heads (e.g., Figure 2 Multiple slice attention modules 205). One advantage of this aspect is that using multiple slice attention heads can improve accuracy and / or reduce computational cost.

[0146] Example Processing System

[0147] Fig. 9 Describes a method that can be configured to perform the present invention such as for Figures 1 to 8 An example processing system 900 of the method is described.

[0148] The processing system 900 includes a central processing unit (CPU) 902, which may be a multi-core CPU in some examples. Instructions executed at the CPU 902 may be loaded, for example, from a program memory associated with the CPU 902, or may be loaded from the memory 924.

[0149] The processing system 900 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 904 , a digital signal processor (DSP) 906 , a neural processing unit (NPU) 908 , a multimedia processing unit 910 , and wireless connectivity components 912 .

[0150] In some aspects, one or more of the CPU 902, GPU 904, DSP 906, and NPU 908 may be configured to perform the operations described herein. Figures 1 to 8 The method described.

[0151] An NPU, such as 908, is typically a dedicated circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0152] NPUs such as 908 may be configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs may be instantiated on a single chip such as a system on a chip (SoC), while in other examples, multiple NPUs may be part of a dedicated machine learning accelerator device.

[0153] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can usually still be performed independently.

[0154] NPUs designed to accelerate training are typically configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (usually labeled or tagged), iterating on the dataset, and then adjusting model parameters (such as weights and biases) to improve model performance. Typically, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0155] NPUs designed to accelerate inference are typically configured to operate on complete models. Such NPUs can thus be configured to input new data slices and quickly process the data through an already trained model to generate model output (e.g., inference).

[0156] In some aspects, the NPU 908 may be implemented as part of one or more of the CPU 902 , GPU 904 , and / or DSP 906 .

[0157] In some aspects, the wireless connectivity component 912 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 912 is further connected to one or more antennas 914.

[0158] The processing system 900 may also include one or more sensor processing units 916 associated with any manner of sensors, one or more image signal processors (ISPs) 918 associated with any manner of image sensors, and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0159] The processing system 900 may also include one or more input and / or output devices 922, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0160] In some examples, one or more processors of processing system 900 may be based on the ARM or RISC-V instruction set.

[0161] The processing system 900 also includes a memory 924 , which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 924 includes computer-executable components that can be executed by one or more of the aforementioned components of the processing system 900 .

[0162] Specifically, in this example, memory 924 includes processing component 924A, slicing component 924B, removing slicing component 924C, executing component 924D, abstracting component 924E, overlapping component 924F, convolution component 924G, embedding component 924H, inference component 924I, and model parameters 924J (e.g., weights, biases, and other machine learning model parameters). One or more of the depicted components and other components not depicted may be configured to perform various aspects of the methods described herein.

[0163] For example, processing component 924A can perform various processing operations, such as normalizing data (e.g., Figure 2 Normalization layers 206 and 210), nonlinear transformations (e.g., via Figure 2 FFN 212), linear operations (e.g., via Figure 3 Linear layer 375) etc.

[0164] Slice component 924B (which may correspond to Figure 3 The slice layer 310 of FIG. 10 is generally used to slice the input sequence, as discussed above. Removing the slice component 924C (e.g., Figure 3 The removed slice layer 370) can typically be used to remove slices to reconstruct a data sequence.

[0165] In some aspects, the execution component 924D can generally be used to perform or calculate various attentions (e.g., via the slice attention layer 208), which can include local attention (e.g., Figure 3 315) and / or global attention (e.g., Figure 3 Section 345).

[0166] Abstract component 924E (which may correspond to Figure 3 The slice embedding element 335) can typically be used to resize the data and / or provide an abstraction (such as via an average pooling operation).

[0167] In some aspects, the overlapping component 924F can be used to provide overlapping local attention, such as via Figure 5 The local attention element 520 and / or Figure 6 In the illustrated example, the convolution component 924G may be used to perform various convolution operations, such as implementing focal local attention, as described above with reference to Figure 7 Embedded component 924H (which may correspond to Figure 2 and Figure 3 The embedding layer 202) can generally be used to generate an embedding of the input data.

[0168] In the illustrated example, the inference component 924I can generally be used to orchestrate one or more of the depicted components to perform inference (e.g., using compound slice attention to generate output inferences). Model parameters 924J generally include any parameters of the model, such as Figure 3 The local attention parameter is 325. Figure 3 The global attention parameters 355, etc.

[0169] Generally, the processing system 900 and / or its components may be configured to perform the methods described herein.

[0170] It is worth noting that in other aspects, aspects of the processing system 900 may be omitted, such as where the processing system 900 is a server computer, etc. For example, in other aspects, the multimedia processing unit 910, the wireless connectivity component 912, the sensor processing unit 916, the ISP 918, and / or the navigation processor 920 may be omitted. Furthermore, aspects of the processing system 900 may be distributed.

[0171] Notice Fig. 9 This is merely one example, and alternative processing systems having fewer, additional, and / or alternative components may be used in other examples.

[0172] Sample Clauses

[0173] Specific implementation examples are described in the following numbered clauses:

[0174] Clause 1: A computer-implemented method, the method comprising: accessing an input data sequence; slicing the input data sequence based on a slice length hyperparameter to generate a stacked slice input data representation; processing the stacked slice input data representation using a slice attention layer to generate a stacked slice output data representation; and deslicing the stacked slice output data representation to generate an output data sequence. An advantage of this aspect is that the sliced ​​attention operation can be performed with reduced computational complexity and / or improved attention output compared to some conventional attention operations.

[0175] Clause 2: A method according to clause 1, wherein processing the stacked slice input data representation using the slice attention layer to generate the stacked slice output data representation includes: processing the stacked slice input data representation using a high-resolution local attention layer to generate local attention output data; processing the local attention output data using a slice embedding layer to generate a slice embedding; processing the slice embedding using a reduced-resolution global attention layer to generate global attention output data; and performing a broadcast addition of the local attention output data and the global attention output data to generate the stacked slice output data representation. One advantage of this aspect is that local attention can be accurately generated using high-resolution local attention, while global attention can be generated at a reduced computational cost using reduced-resolution global attention.

[0176] Clause 3: A method according to clause 2, wherein: processing the stacked slice input data representation with the high-resolution local attention layer comprises applying a first set of training weights to the stacked slice input data representation, and processing the slice embedding with the reduced-resolution global attention layer comprises applying a second set of training weights to the slice embedding. An advantage of this aspect is that the local and global attention layers can use different sets of training weights, which can improve model performance.

[0177] Clause 4: The method of any one of clauses 2 to 3, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises: generating a local key vector, a local query vector, and a local value vector by applying the first set of training weights to the stacked slice input data representation; and generating the local attention output data based on the local key vector, the local query vector, and the local value vector. An advantage of this aspect is that the local attention can be generated using weights learned during training for high-resolution local attention.

[0178] Clause 5: A method according to any one of clauses 2 to 4, wherein: processing the stacked slice input data representation using the high-resolution local attention layer also includes adding local position embeddings to the local key vector and the local query vector, and the length of the local position embedding is based on the slice length hyperparameter. An advantage of this aspect is that the position embedding can be customized to take into account local positioning based on the slice.

[0179] Clause 6: A method according to any of clauses 2 to 5, wherein processing the slice embedding with the reduced-resolution global attention layer comprises: generating a global key vector, a global query vector, and a global value vector by applying the second set of training weights to the slice embedding; and generating the global attention output data based on the global key vector, the global query vector, and the global value vector. An advantage of this aspect is that global attention can be generated using weights learned during training for reduced-resolution global attention.

[0180] Clause 7: A method according to any one of clauses 2 to 6, wherein: processing the slice embedding with the reduced resolution global attention layer comprises adding a global position embedding to the global key vector and the global query vector, and the length of the global position embedding is based on the input data sequence length divided by the slice length hyperparameter. An advantage of this aspect is that the position embedding can be customized to take into account global positioning.

[0181] Clause 8: The method of any one of claims 2 to 7, wherein processing the stacked slice input data representation with the high resolution local attention layer comprises performing overlapping slice local attention. An advantage of this aspect is that overlapping slice local attention can reduce or prevent context fragmentation.

[0182] Clause 9: The method of clause 8, wherein slicing the input data sequence is further performed based on an overlap hyperparameter to generate overlapping slices of the input data sequence. An advantage of this aspect is that overlapping slices can improve model accuracy.

[0183] Clause 10: A method according to any one of clauses 2 to 9, wherein processing the stacked slice input data representation with the high-resolution local attention layer comprises performing focal slice local attention. An advantage of this aspect is that focal slice local attention can reduce or eliminate context fragmentation.

[0184] Clause 11: The method of clause 10, wherein: slicing the input data sequence comprises generating a plurality of slices having a plurality of sequence lengths, and performing the focal slice local attention comprises: generating a plurality of intermediate tensors based on the plurality of slices; and aggregating the plurality of intermediate tensors. An advantage of this aspect is that aggregating intermediate tensors can reduce computational costs.

[0185] Clause 12: A method according to any one of clauses 1 to 10, wherein the slice attention layer comprises a plurality of slice attention heads. An advantage of this aspect is that using multiple slice attention heads may improve accuracy and / or reduce computational cost.

[0186] Clause 13: A processing system, the processing system comprising: a memory, the memory comprising computer-executable instructions; and one or more processors, the one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of clauses 1 to 12.

[0187] Clause 14: A processing system comprising: means for performing the method according to any one of clauses 1 to 12.

[0188] Clause 15: A non-transitory computer readable medium comprising computer executable instructions which, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 12.

[0189] Clause 16: A computer program product embodied on a computer readable storage medium, the computer program product comprising: code for performing the method according to any one of clauses 1 to 12.

[0190] Additional considerations

[0191] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Each example may ignore, replace or add each process or component as appropriate. For example, the described method may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described for some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functions, or structures and functions that are supplemented or substituted for the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0192] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0193] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0194] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Moreover, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. In addition, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0195] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any appropriate component that can perform the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0196] The following claims are not intended to be limited to the various aspects shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, unless otherwise specified, the reference to the singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specified, the term "some" refers to one or more. No claim element should be interpreted according to the provisions of 35 U.S.C. § 112 (f) unless the element is explicitly recorded using the phrase "a component for...", or in the case of a method claim, the phrase "a step for..." is used to record the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to a person of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A computer-implemented method, the method comprising: Access input data sequences; Slicing the input data sequence based on a slice length hyperparameter to generate a stacked slice input data representation; processing the stacked slice input data representation using a slice attention layer to generate a stacked slice output data representation; as well as The stacked slice output data representation is de-sliced ​​to generate an output data sequence.

2. The method of claim 1 , wherein processing the stacked slice input data representation using the slice attention layer to generate the stacked slice output data representation comprises: processing the stacked slice input data representation using a high-resolution local attention layer to generate local attention output data; Processing the local attention output data using a slice embedding layer to generate a slice embedding; processing the slice embedding using a reduced-resolution global attention layer to generate global attention output data; as well as Perform a broadcast addition of the local attention output data and the global attention output data to generate the stacked slice output data representation.

3. The method according to claim 2, wherein: Processing the stacked slice input data representation with the high-resolution local attention layer includes applying a first set of trained weights to the stacked slice input data representation, and Processing the slice embedding with the reduced-resolution global attention layer includes applying a second set of trained weights to the slice embedding.

4. The method of claim 3, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises: generating a local key vector, a local query vector, and a local value vector by applying the first set of training weights to the stacked slice input data representation; as well as The local attention output data is generated based on the local key vector, the local query vector and the local value vector.

5. The method according to claim 4, wherein: Processing the stacked slice input data representation with the high-resolution local attention layer further comprises adding local position embeddings to the local key vector and the local query vector, and The length of the local position embedding is based on the slice length hyperparameter.

6. The method of claim 3, wherein processing the slice embedding using the reduced-resolution global attention layer comprises: generating a global key vector, a global query vector, and a global value vector by applying the second set of training weights to the slice embedding; as well as The global attention output data is generated based on the global key vector, the global query vector, and the global value vector.

7. The method according to claim 6, wherein: Processing the slice embedding with the reduced-resolution global attention layer includes adding a global position embedding to the global key vector and the global query vector, and The length of the global position embedding is based on the input data sequence length divided by the slice length hyperparameter.

8. A method according to claim 2, wherein processing the stacked sliced ​​input data representation using the high-resolution local attention layer includes performing overlapping slice local attention, and wherein slicing the input data sequence is further performed based on an overlapping hyperparameter to generate overlapping slices of the input data sequence.

9. The method of claim 2, wherein processing the stacked slice input data representation with the high-resolution local attention layer comprises performing focal slice local attention, wherein: Slicing the input data sequence includes generating a plurality of slices having a plurality of sequence lengths; and Performing the focal slice local attention includes: generating a plurality of intermediate tensors based on the plurality of slices; as well as The plurality of intermediate tensors are aggregated.

10. The method of claim 1, wherein the slice attention layer comprises a plurality of slice attention heads.

11. A processing system, the processing system comprising: a memory including computer executable instructions; and One or more processors configured to execute the computer-executable instructions to cause the processing system to perform operations including: Access input data sequences; Slicing the input data sequence based on a slice length hyperparameter to generate a stacked slice input data representation; processing the stacked slice input data representation using a slice attention layer to generate a stacked slice output data representation; as well as The stacked slice output data representation is de-sliced ​​to generate an output data sequence.

12. The processing system of claim 11 , wherein processing the stacked slice input data representation using the slice attention layer to generate the stacked slice output data representation comprises: processing the stacked slice input data representation using a high-resolution local attention layer to generate local attention output data; Processing the local attention output data using a slice embedding layer to generate a slice embedding; processing the slice embedding using a reduced-resolution global attention layer to generate global attention output data; as well as Perform a broadcast addition of the local attention output data and the global attention output data to generate the stacked slice output data representation.

13. The processing system of claim 12, wherein: Processing the stacked slice input data representation with the high-resolution local attention layer includes applying a first set of trained weights to the stacked slice input data representation, and Processing the slice embedding with the reduced-resolution global attention layer includes applying a second set of trained weights to the slice embedding.

14. The processing system of claim 13, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises: generating a local key vector, a local query vector, and a local value vector by applying the first set of training weights to the stacked slice input data representation; as well as The local attention output data is generated based on the local key vector, the local query vector and the local value vector.

15. The processing system of claim 14, wherein: Processing the stacked slice input data representation with the high-resolution local attention layer further comprises adding local position embeddings to the local key vector and the local query vector, and The length of the local position embedding is based on the slice length hyperparameter.

16. The processing system of claim 13, wherein processing the slice embedding using the reduced-resolution global attention layer comprises: generating a global key vector, a global query vector, and a global value vector by applying the second set of training weights to the slice embedding; as well as The global attention output data is generated based on the global key vector, the global query vector, and the global value vector.

17. The processing system of claim 16, wherein: Processing the slice embedding with the reduced-resolution global attention layer includes adding a global position embedding to the global key vector and the global query vector, and The length of the global position embedding is based on the input data sequence length divided by the slice length hyperparameter.

18. The processing system of claim 12, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises performing overlapping slice local attention.

19. The processing system of claim 12, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises performing focal slice local attention.

20. The processing system of claim 11, wherein the slice attention layer comprises a plurality of slice attention heads.

21. A non-transitory computer readable medium comprising computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform operations comprising: Access input data sequences; Slicing the input data sequence based on a slice length hyperparameter to generate a stacked slice input data representation; processing the stacked slice input data representation using a slice attention layer to generate a stacked slice output data representation; as well as The stacked slice output data representation is de-sliced ​​to generate an output data sequence.

22. The non-transitory computer-readable medium of claim 21 , wherein processing the stacked slice input data representation using the slice attention layer to generate the stacked slice output data representation comprises: processing the stacked slice input data representation using a high-resolution local attention layer to generate local attention output data; Processing the local attention output data using a slice embedding layer to generate a slice embedding; processing the slice embedding using a reduced-resolution global attention layer to generate global attention output data; as well as Perform a broadcast addition of the local attention output data and the global attention output data to generate the stacked slice output data representation.

23. The non-transitory computer readable medium of claim 22, wherein: Processing the stacked slice input data representation with the high-resolution local attention layer includes applying a first set of trained weights to the stacked slice input data representation, and Processing the slice embedding with the reduced-resolution global attention layer includes applying a second set of trained weights to the slice embedding.

24. The non-transitory computer-readable medium of claim 23, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises: generating a local key vector, a local query vector, and a local value vector by applying the first set of training weights to the stacked slice input data representation; as well as The local attention output data is generated based on the local key vector, the local query vector and the local value vector.

25. The non-transitory computer readable medium of claim 24, wherein: Processing the stacked slice input data representation with the high-resolution local attention layer further comprises adding local position embeddings to the local key vector and the local query vector, and The length of the local position embedding is based on the slice length hyperparameter.

26. The non-transitory computer-readable medium of claim 23, wherein processing the slice embedding using the reduced-resolution global attention layer comprises: generating a global key vector, a global query vector, and a global value vector by applying the second set of training weights to the slice embedding; as well as The global attention output data is generated based on the global key vector, the global query vector, and the global value vector.

27. The non-transitory computer readable medium of claim 26, wherein: Processing the slice embedding with the reduced-resolution global attention layer includes adding a global position embedding to the global key vector and the global query vector, and The length of the global position embedding is based on the input data sequence length divided by the slice length hyperparameter.

28. The non-transitory computer-readable medium of claim 22, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises performing overlapping slice local attention.

29. The non-transitory computer-readable medium of claim 22, wherein processing the stacked slice input data representation using the high-resolution local attention layer comprises performing focal slice local attention.

30. A processing system, the processing system comprising: Components for accessing input data sequences; means for slicing the input data sequence based on a slice length hyperparameter to generate a stacked slice input data representation; means for processing the stacked slice input data representation using a slice attention layer to generate a stacked slice output data representation; as well as Means for slicing the stacked slice output data representation to generate an output data sequence.

Images