Long Sequence Modeling via State Space Model (SSM) Enhanced Transformers

Abstract

A computing device is provided that includes a processor configured to execute a transformer including an encoder having a global layer configured to receive a tokenized embedding of each of a plurality of tokens in a local input sequence and to calculate a global self-attention vector for each of the tokenized embeddings. The encoder further includes a local layer configured to receive each global self-attention vector from the global layer and to calculate a local self-attention for each local input sequence, and to add the local self-attention vector to the global self-attention vector and normalize it to thereby generate an encoder representation including a self-attention vector for each local input sequence that includes both a global self-attention value and a local self-attention value. The transformer is configured to output a prediction for the global input sequence based on the encoder representation of each of the local input sequences of the global input sequence.

Description

[Background Technology]

[0001] background A Transformer is a machine learning model that uses a self-attention mechanism to encode a sequence of input tokens into an attention vector. Transformers can be configured for a variety of prediction tasks, including, for example, sequence-to-sequence or sequence-to-classification. A sequence-to-sequence Transformer includes a decoder that decodes the attention vector into an output sequence of tokens. Applications of sequence-to-sequence Transformer models include language models configured to convert a sequence of words from a source language to a target language and language models that predict the next word in an input word sequence. Applications of sequence-to-classification models include, for example, sentiment analysis models configured to predict the sentiment (positive, neutral, negative, etc.) of a sequence of text. The Transformer's self-attention mechanism has been found to provide improved performance of such language and sentiment models over bidirectional recurrent neural networks due to the self-attention mechanism's ability to equally attend to any other token in the input sequence, compared to, for example, bidirectional recurrent neural networks, where attention between two tokens in a sequence would be attenuated as the distance between the tokens increases. One drawback of this type of transformer is that the computational complexity of computing self-attention in this way is quadratic in time and memory space based on the length of the input sequence, thereby imposing practical limits on the length of the token sequence analyzed. Contemporary transformers limit input sequences to, for example, 512 or 1024 tokens. Another challenge with transformers that apply full self-attention is that they can easily overfit, making them prone to learning from noise. A family of transformers has been developed that limits full attention mechanisms to linear complexity using fast algorithms, but even these can suffer from overfitting due to the lack of structural bias.To address this, transformers with partial attention mechanisms such as sparse attention and clustering have been proposed, but these structural biases fail to capture truly global attention and are instead limited to the specific clustering or sparsity regime imposed.

[0002] State-space models are a type of model that can capture global attention, but traditional state-space models are primarily based on recurrent neural networks, and these traditional recurrent neural network state-space models cannot compute the dependency between any two input tokens in a sequence in an equally effective way as attention-based transformer models do.

[0003] Therefore, there is an opportunity to improve the performance of attention-based models that make predictions based on long input sequences. Summary of the Invention

[0004] overview A computing device is provided that includes a processor configured to execute a transformer including an encoder having a global layer configured to receive a tokenized embedding of each of a plurality of tokens in a local input sequence and calculate a global self-attention vector for each of the tokenized embeddings. The encoder further includes a local layer configured to receive each global self-attention vector from the global layer, calculate a local self-attention for each local input sequence, and add the local self-attention vector to the global self-attention vector and normalize it to thereby generate an encoder representation including a self-attention vector for each local input sequence that includes both a global self-attention value and a local self-attention value. The transformer is configured to output a prediction for the global input sequence based on the encoder representation of each of the local input sequences of the global input sequence. This summary is provided to introduce selected concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any shortcomings noted in any part of this disclosure. [Brief explanation of the drawings]

[0005] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] We show the performance of the conventional transformer with full attention, the conventional transformer with windowed attention, and the S4 model. [Figure 2] 1 illustrates a computing system including a computing device configured to perform long sequence modeling via an SSM-enhanced transformer having a global layer including a state-space model according to an example of the present disclosure. [Figure 3]3 shows details of the global layer of the decoder of the transformer of FIG. 2. [Figure 4] Window attention and chunk attention are shown. [Figure 5] We present experimental results on the Long Range Arena (LRA) dataset, comparing our version of the transformer model with other conventional models. [Figure 6] We present experimental results benchmarking three versions of our transformer model against five other conventional models on the Wikitext-103 dataset. [Figure 7] We present experimental results of benchmarking two versions of our transformer model against two other conventional models on the GLUE development set. [Figure 8] We present a comparison of two versions of our model with four previous models in terms of memory usage and number of updates per second (computational complexity) for three different global sequence lengths. [Figure 9] 1 shows a flowchart of a computerized method according to an exemplary implementation of the present disclosure. [Figure 10] 3 shows a schematic diagram of an exemplary computing environment in which the computing device of FIG. 2 may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0006] Detailed Description Transformer models have achieved excellent performance in various natural language processing tasks. However, the quadratic computational cost of attention mechanisms limits their practicality for long sequences. Conventional attention variants sacrifice the Transformer model's ability to effectively compute global information to improve computational efficiency. On the other hand, state-space models (SSMs) are well-suited to long sequences but are not flexible enough to capture complex local information. To address this challenge, an SSM-enhanced Transformer model is presented. Specifically, an SSM is incorporated into the input layer of the Transformer model's encoder, and efficient local attention methods are employed for other layers. The SSM captures global information, which complements the lack of long-range dependency issues in local attention methods. Experimental results discussed below on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the disclosed method. Furthermore, the disclosed system and method are used to pre-train a sequence-to-sequence Transformer model, and fine-tuning results on natural language understanding and natural language generation tasks are presented.

[0007] 1. Introduction Transformer models have achieved excellent performance on various natural language processing tasks, such as language modeling, natural language generation, and natural language understanding. These models utilize an attention mechanism that calculates dependency scores for every pair of tokens in an input sequence. Therefore, full attention has quadratic time and space complexity with respect to the sequence length. However, such complexity is computationally prohibitive for tasks involving long sequences, such as text summarization and question answering. For example, a Transformer model with 250M parameters was found to consume over 80G of GPU memory when the sequence length was 8k.

[0008] In addition, Transformer models with full attention are prone to overfitting due to the lack of structural bias. That is, the attention mechanism does not assume any structural a priori information about the input. For example, order information (e.g., through sinusoidal encoding) is required to train the model. Therefore, full attention is too flexible, and as a result, the Transformer model can easily overfit to the noise contained in the input sequence. This significantly limits the model's practicality in long sequence modeling, where the dependent signal is often weak and the signal-to-noise ratio is often low. For a two-way classification task, an exemplary Transformer with a full attention mechanism was experimentally found to have an accuracy of 57.5%, which is nearly 30% lower than prior art methods with strong structural bias (see Table 1 in Figure 5 for details).

[0009] Various approaches have been proposed to reduce the quadratic complexity discussed above and / or introduce structural biases that are lacking in the full attention mechanism. For example, in approximation methods, full attention is approximated using fast algorithms with linear complexity. For example, the attention score matrix (i.e.,

number

[0010] Partial attention methods have been proposed to incorporate structural biases into Transformer models. Traditional partial attention methods can be classified into sparse attention methods and clustering methods. In sparse attention, each token only pays attention to a subset of all tokens according to a predetermined sparsity pattern. In clustering methods, tokens are divided into several clusters and only inter-cluster attention is performed. However, the structural biases introduced by these methods limit the model's ability to capture global information. For example, in local window attention, each token is assumed to depend only on its neighboring tokens, which inevitably results in the loss of long-distance and global information.

[0011] Contrary to partial attention, state-space models (SSMs) introduce different structural biases that are adapted to compute global information. Specifically, SSM designs roughly fix global dependency patterns, facilitating effective and efficient computation. These models can be viewed as linear recurrent neural networks with specially designed roughly fixed weights. Furthermore, efficient linear-time and space-complexity algorithms for training such models have been developed using conventional techniques. However, the structural biases introduced by these algorithms remain limited in that SSMs are not sophisticated enough to capture local information. This is because, unlike attention, SSMs do not explicitly compute dependencies between input tokens.

[0012] To address these challenges, we propose a hierarchically structured multi-layer Transformer model that can effectively and efficiently capture complex dependencies. Specifically, SSM is incorporated into the input layer of the Transform Decoder model, so that after this layer, the input is integrated with global information. Because SSM only provides coarse global information, a subsequent top layer of the disclosed embodiment employs a sparse attention variant to capture refined, more complex local information. In other words, SSM serves as a strong structural bias that incorporates global information, complementing the lack of long-range dependency challenges in sparse attention methods.

[0013] As discussed in detail below, test result data demonstrate the efficiency and effectiveness of the disclosed system and method for various natural language processing tasks. First, the proposed system and method is shown to outperform existing methods on the Long Range Arena benchmark, which is designed to test a model's ability to model long sequences. Second, data is presented showing that the present system and method is not only significantly faster than conventional Transformers, but also provides better performance in autoregressive language modeling. Third, data from language model pre-training and fine-tuning experiments is presented. Specifically, pre-trained sequence-to-sequence Transformer models are fine-tuned for various tasks, including natural language understanding and natural language generation benchmarks. In all these settings, the present system and method outperforms conventional pre-trained networks, such as T5 and Long T5, a variant of T5 adapted for long sequence modeling. Finally, data from analysis and ablation experiments is presented to further demonstrate the effectiveness of the disclosed system and method.

[0014] 2. Preliminary Essay 2.1 Attention Mechanism If the input to the layer is

number

number

number

number

[0015] 2.2 State Space Model Continuous-time state-space model. A continuous-time latent space model is a model that expresses a one-dimensional input signal u(t) as s The first step is to map a multi-dimensional potential state x(t) into a one-dimensional output signal y(t), which is then mapped to a one-dimensional output signal y(t). x'(t)=Ax(t)+Bu(t),y(t)=Cx(t) (2) where, A∈

number

number

number

[0016] Equation 2 can be utilized to model long sequences. Because randomly initialized parameters A, B, and C cannot well model long-range dependencies, a class of matrices (called HiPPO, Higher Order Polynomial Projection Operator) is proposed to initialize A. HiPPO matrices are designed such that the state x(t) can remember the history of the input u(t) up to time t.

[0017] Discrete-time state-space models. In practice, natural language inputs (u0,u1,...,u) that cannot be easily modeled by continuous-time state-space models L ) (where L is the sequence length) is used. To facilitate modeling such discrete data, the model in Equation 2 can be discretized by a step size Δ (using bilinearization methods), resulting in:

number

[0018] After the above recursive expression is expanded, we get:

number

[0019] This is a convolutional representation

number

number

[0020] In Equation 4, the output y is the convolution kernel

number

[0021] Structured State Space Sequence Model (S4): The structured state space sequence model S4 was developed to efficiently calculate Equation 4. Specifically, B and C in Equation 2 are initialized randomly, and A is initialized as follows:

number

[0022] The convolution kernel in Equation 4 can then be efficiently computed with linear O(L) computation time and memory space complexity.

[0023] 3. Method First, data from simulations is presented to demonstrate that SSM does not model local information well. Then, the disclosed system and method is discussed, which efficiently and effectively combines global and local information by incorporating SSM into a transformer architecture.

[0024] 3.1 Attention vs. State Space Model Here, the S4 model is compared with an exemplary transformer with a full attention mechanism and a transformer configured with a window attention mechanism. In window attention, each token can only pay attention to its neighboring tokens within a fixed-size window (see Figure 4 for details). Figure 4 shows window attention (left) and chunk attention (right). Simulation experiments were conducted for token-level language modeling. In this setting, local information is more important than global information. This is because in practice, it is rare to see words (tokens) with thousands of positions exhibiting strong dependencies separately.

[0025] The experimental results are shown in Figure 1. We can see that the Transformer with full attention and window attention outperforms the S4 model. Note that replacing full attention with window attention does not significantly impair model performance, indicating that local information is more important in this setting. State-space models such as the S4 model generate nearly fixed dependency patterns (e.g., the convolution kernel in Equation 4). Furthermore, unlike traditional self-attention mechanisms, SSM does not directly compute dependencies between tokens. Therefore, SSM is not sophisticated enough to capture local information and, as a result, performs poorly for language modeling tasks.

[0026] 3.2 SSM enhanced transformer Here, a system and method according to the present disclosure is described that utilizes a multi-layer Transformer model that can capture complex global and local information. The overall architecture is shown in FIG. 2. The proposed model adopts a hierarchical structure. In particular, at the lowest layer of the model (called the global layer), global dependencies are captured using SSM. Because the SSM provides only coarse global information, subsequent local layers facilitate the model to handle more sophisticated and complex local dependencies. In other words, the SSM acts as a strong structural bias that incorporates global information into the input.

[0027] To instantiate the local layer, the full attention in the traditional transformer layer is replaced with an off-the-shelf, efficient sparse attention method. The present system and method are flexible enough to accommodate various methods such as window attention and chunk attention. See Figure 3 for an illustration of the portions of the global sequence (shown in gray) to which window attention and chunk attention are directed.

[0028] In the global layer (Figure 3), when an input X is given to the global layer, the output Y is calculated as follows: X local =Local(LN(X)), X global =SSM(LN(X)), X a =W[LN(X local ),LN(X global )]+X, Y=FEN(LNX a ))+X a where LN(·) represents layer normalization (Ba et al., 2016), FFN(·) represents a two-layer feedforward neural network, and W is a trainable weight that combines the local and global representations. Layer normalization is performed by adjusting X to align their scales. local and X global Note that this applies to In this study, S4 is chosen as the state space model.

[0029] FIG. 2 illustrates a computing system 100 including a computing device 10 configured to perform long sequence modeling via an SSM-enhanced transformer having a global layer 22 including a state-space model. The computing device 10 includes a processor 12 and associated working memory 14 (e.g., RAM) and non-volatile memory 16 (e.g., solid-state drive (SSD)). The associated working memory 14 and non-volatile memory 16 include stored instructions that, when executed by the processor 12, cause the processor 12 to implement software for implementing the functions described herein. The processor 12 is configured to execute a pre-processing module 30 including a tokenization layer 32 and an embedding layer 34. The tokenization layer 32 tokenizes input data 40 into a global sequence of tokens. For example, the input data 40 may be text data, image data (including still images or video data), audio data, or other data. The processor 12 is further configured to execute a transformer 18 including an encoder 20. The encoder 20 has a global layer 22 and at least one or more local layers 24. Although three local layers 24 are shown in the depicted configuration, it will be understood that this is merely exemplary. In one example, fifteen local layers 24 are provided.

[0030] It is recognized that the input data may include text or other types of sequenced data that are tokenized by the tokenization layer into tokens for a global input sequence. Embeddings for the tokens are generated by the embedding layer 34, thereby generating a tokenized embedding for the global input sequence. The tokenized embedding global input sequence is then split into multiple local input sequences by a preprocessing module and passed to the global layer 22 of the encoder of the transformer 18. The global layer 22 is configured to receive, for each of the multiple local input sequences in the global input sequence, the tokenized embedding of each of the multiple tokens in the local input sequence from the embedding layer 34 and calculate a global self-attention vector for each of the tokenized embeddings in the local input sequence. The encoder of the Transformer 18 further includes a local layer 24 configured to receive a global self-attention vector for each local input sequence from the global layer 22, calculate a local self-attention for the local input sequence, and add the local self-attention vector to the global self-attention vector and normalize it, thereby generating an encoder representation including a self-attention vector for each local input sequence that includes both a global self-attention value and a local self-attention value. The Transformer 18 is configured to output a prediction 50 for the global input sequence based on the encoder representation of each of the local input sequences of the global input sequence according to a prediction task.

[0031] The Transformer 18 may also include a classification layer configured to receive the encoder representations and generate predictions 50, where the predictions 50 include one or more predicted classifications. In such a configuration, the Transformer 18 is configured as a sequence to classification Transformer model, an example of which is a sentiment analysis model for text input. Alternatively, the Transformer 18 may be a sequence-to-sequence Transformer that includes a decoder that includes one or more local layers 24 and a global layer 22. In such a configuration, the decoder is configured to decode and receive the encoder representations and generate an output sequence of tokens as a prediction based on multiple local input sequences within a global input sequence.

[0032] Figure 3 shows the architecture and data flow details of the global layer 22 of the encoder of Figure 1. As shown in Figure 3, the global layer 22 includes a state-space model layer 60 configured to receive a tokenized embedding of each of a plurality of tokens in the local input sequence from the embedding layer 34. The state-space model layer 60 may include a discrete-time structured state-space sequence model parameterized by a normal matrix plus a low-rank matrix. An example of such a model is the S4 model. Using such a model, computing global self-attention using the global layer 22 including the state-space model layer 60 can be achieved with linear computational complexity and linear memory complexity relative to the global input sequence.

[0033] Continuing with Figure 3, the global layer 22 further includes a local layer 24 located in a parallel data path to the state space model layer 60. The local layer 24 is configured to receive a tokenized embedding for each of the multiple tokens in the local input sequence from the embedding layer 34 and to compute a local self-attention for the local input sequence. After each of the SSM layer and the local layer 24, a respective normalization layer is provided for normalizing the values ​​in the respectively computed attention vector in each data flow path.

[0034] As shown in Figure 3, the global layer 22 further includes a synthesis layer 66 configured to concatenate the global and local self-attention computed in the global layer 22. The global layer 22 further includes a summing and normalizing layer 68A configured to add and normalize the tokenized embeddings output from the embedding layer 34 to the concatenated global and local self-attention computed in the global layer 22. Finally, the global layer 22 further includes a feedforward network 70 configured to receive the normalized synthesized global and self-attention vectors computed in the global layer 22 and output the predicted global layer output at inference time. Another summing and normalizing layer 68B typically follows the feedforward network, as shown.

[0035] 4. Experimental Results In the experiments described below, all of the models were implemented using PyTorch, Fairseq, and HuggingFace.

[0036] 4.1 Long Range Arena Datasets. The effectiveness of the proposed model is evaluated on the Long Range Arena, a benchmark adapted to evaluate a model's ability to model long sequences. The benchmark includes six tasks: ListOps (testing the ability to model hierarchically structured data), byte-level text classification on the IMDB movie reviews dataset, byte-level document retrieval on the ACL anthology network, pixel-level image classification on C1FAR-10, Pathfinder (testing the ability to model spatial dependencies), and a longer version of Pathfinder: Path-X.

[0037] Models. Following standard settings, small models (e.g., less than 2M parameters) are used for all tasks in these experiments. The computational budget is constrained so that all models are trained at a similar rate during the same amount of time.

[0038] To aggregate local information, the following two methods were considered: window attention and chunk attention. For window attention, traditional softmax attention is sparsified, and for chunk attention, MEGA, which adopts gated attention technique, is sparsified. For window attention, the window size was set to 128 except for Path-X, where the window size was set to 1024. For chunk attention, the chunk size was set to 128 except for Path-X, where the chunk size was set to 4096.

[0039] Results. The experimental results are summarized in Table 1. We can see that both variants of our model (softmax window and MEGA chunk), built according to the system shown in Figures 2 and 3, outperform all baselines in terms of average accuracy. For example, the window attention variant outperforms the best-performing baseline (MEGA chunk) by only 0.5%, and the chunk attention variant has a performance gain of 2.4%. Therefore, our approach is better suited to modeling long sequences than conventional approaches.

[0040] 4.2 Language Modeling The model is further evaluated by conducting language modeling experiments on the Wikitext-103 dataset. This dataset contains English Wikipedia articles, and the total number of training tokens is 103M. All experiments use a large-scale Transformer model with 16 layers and approximately 250M parameters. The input sequence length is set to 3k and trained for a total of 286k steps. Similar to the Long Range Arena (LRA) experiments, the model employs either window attention (softmax window) or chunk attention (MEGA chunk) as the local information extractor. Additionally, another variant, FLASH chunk, is evaluated. Here, FLASH is a gated attention method similar to MEGA, but with sparsification.

[0041] The experimental results are presented in Table 2 and reproduced in Figure 6. These results show that by combining global and local information, the proposed model achieves significant performance improvement and outperforms other baselines. For example, traditional window attention has a complexity of 19.7 on the test set, and by incorporating SSM into our model, a complexity gain of 1.2 is achieved. Our model (with softmax window) is not only significantly faster than the Transformer with full attention, but also yields better performance.

[0042] 4.3 Pre-training large-scale language models It is recognized that the present model can be applied to pre-training large-scale language models. For example, this pre-training can be of a large-scale language model on the Wikipedia dataset and the BookCorpus dataset, or both of these plus the common crawl news dataset CC-News. These are very large datasets, and pre-training a Transformer language on such datasets can often take weeks of GPU time; therefore, the speedup in computational complexity and low memory requirements of the present model will have a significant saving in the time and cost of such pre-training.

[0043] 5.Memory usage and number of updates per second Figure 8 shows the memory usage and number of updates per second of our transformer using the window attention and MEGA techniques discussed above compared to conventional transformers for various global sequence lengths of 3K, 4K, and 6K. As can be seen, especially for the longest sequences, our model with either the window or MEGA configuration achieves the highest number of updates per second (evidence of lowest computational complexity / cost) while using the lowest amount of memory.

[0044] 6. Appendix 6.1 Efficient Transformer Models In Equation 1, Q, K, V∈

number

[0045] One approach is to adopt sparse attention. That is, each token pays attention only to a subset of all tokens (e.g., neighboring tokens within a fixed-size window) according to a predetermined pattern. Some examples include Sparse Transformer, BlockBERT, Longformer, ETC, BigBird, HEPOS, and Poolingformer.

[0046] Another approach is to use low-rank projection. For example, in Linformer, the attention mechanism of Equation 1 is

Number

Number

Number

[0047] Kernel-based methods can be used to approximate full attention Attn(X). In these methods, second-order time softmax attention is replaced by a final linear-time kernel approximation (e.g., Gaussian and arccosine kernels). Some examples include Linear Transformer, Performer, Random Feature Attention, and FMMformer. Both low-rank projection and kernel-based methods approximate full attention, and therefore, the methods often face non-negligible approximation errors.

[0048] A clustering-based approach can be adopted, where Q or K is divided into several clusters and only inter-cluster attention is performed. Such methods include Reformer, Cluster-former, Sinkhorn Transformer, Fast Transformer, Routing Transformer, and FLASH.

[0049] 6.2 Pre-trained Language Models Pre-trained language models have achieved state-of-the-art performance on various natural language processing tasks. However, most of these models are not suitable for long sequences. For example, BERT uses fixed-length positional embeddings and, as a result, cannot handle sequences with lengths greater than 512. In contrast, LongT5 facilitates training for long sequences by leveraging relative positional embeddings and local window attention. This model targets long sequence modeling tasks such as text summarization and question answering.

[0050] 9 illustrates a flowchart of a computerized method 300 according to an exemplary implementation of the present disclosure. At step 302, the method may include, for each of a plurality of local input sequences within a global input sequence, receiving, at a global layer of a transformer, from an embedding layer, a tokenized embedding of each of a plurality of tokens in the local input sequence. At step 304, the method may further include receiving, at a state-space model layer, from the embedding layer, a tokenized embedding of each of the plurality of tokens in the local input sequence, the global layer including the state-space model layer. At step 306, the method may include computing, at the global layer, a global self-attention vector for each of the tokenized embeddings in the local input sequence. At step 308, the method may include receiving, at the local layer, a global self-attention vector for each local input sequence from the global layer, and computing, at the local layer, a local self-attention for the local input sequence. At step 310, the method further includes receiving, from the embedding layer, a tokenized embedding of each of the plurality of tokens in the local input sequence at a local layer, and computing local self-attention for the local input sequence, the global layer further including a local layer located in a parallel data path to the state space model layer. At step 312, the method may include concatenating the global self-attention and the local self-attention computed in the global layer at a synthesis layer, the global layer further including a synthesis layer.

[0051] At step 314, the method may include adding the local self-attention vector to the global self-attention vector and normalizing it to generate an encoder representation including a self-attention vector for each local input sequence, the self-attention vector including both global and local self-attention values. At step 316, the method may further include adding the tokenized embedding output from the embedding layer to the combined global and local self-attention calculated in the global layer and normalizing it in a summing and normalizing layer, where the global layer further includes a summing and normalizing layer. At step 318, the method may include outputting a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence. At step 320, the method may further include receiving the normalized combined global and self-attention vector calculated in the global layer and output during prediction in a feedforward network, and outputting a predicted global layer output from the feedforward network, where the global layer further includes a feedforward network.

[0052] As the present experimental results demonstrate, it will be appreciated that the above-described systems and methods have potential technical benefits, providing performance advantages in terms of lower computational complexity and memory requirements, as well as higher accuracy predictions for some prediction tasks involving longer sequences of input data than the approaches discussed above.

[0053] 7. Computing environment In some embodiments, the methods and processes described herein may involve computing systems of one or more computing devices. In particular, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0054] 10 illustrates generally a non-limiting embodiment of a computing system 600 that may implement one or more of the methods and processes described above. The computing system 600 is shown in simplified form. The computing system 600 may embody the computing device 10 described above and shown in FIG. 2. The computing system 600 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0055] Computing system 600 includes a logical processor 602, a volatile memory 604, and a non-volatile storage device 606. Computing system 600 may optionally include a display subsystem 608, an input subsystem 610, a communication subsystem 612, and / or other components not shown in FIG. 10 . Logical processor 602 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement data types, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

[0056] A logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of logical processor 602 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel, and / or distributed processing. Individual components of a logical processor may optionally be distributed among two or more separate devices that may be remotely located and / or configured for cooperative processing. Aspects of a logical processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In such cases, it is understood that these virtualized aspects are executed on various physical logical processors of various machines.

[0057] Non-volatile storage device 606 includes one or more physical devices configured to hold instructions executable by the logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage device 606 may be transformed (e.g., to hold various data).

[0058] The non-volatile storage device 606 may include removable and / or internal physical devices. The non-volatile storage device 606 may include optical memory (e.g., CD, DVD, HD DVD, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.) or other mass storage device technologies. The non-volatile storage device 606 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content accessible devices. It will be appreciated that the non-volatile storage device 606 is configured to retain instructions even when power to the non-volatile storage device 606 is disconnected.

[0059] Volatile memory 604 may include physical devices including random access memory. Volatile memory 604 is typically utilized by logical processor 602 to temporarily store information during the processing of software instructions. It is recognized that volatile memory 604 typically does not continue to store instructions once power to volatile memory 604 is removed.

[0060] Aspects of the logic processor 602, volatile memory 604, and non-volatile storage device 606 may be integrated together into one or more hardware logic components, which may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASICs / ASICs), program and application specific standard products (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0061] The terms “module,” “program,” and “engine” may be used to describe an aspect of computing system 600 that is typically implemented in software by a processor to perform a particular function using a portion of volatile memory. This function involves a translation operation that specifically configures the processor to perform this function. Thus, a module, program, or engine may be instantiated via logical processor 602, which executes instructions held by non-volatile storage device 606 using a portion of volatile memory 604. It will be understood that various modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by various applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual executable files, data files, libraries, drivers, scripts, database records, etc., or groups thereof.

[0062] If included, the display subsystem 608 may be used to present a visual representation of the data maintained by the non-volatile storage device 606. The visual representation may take the form of a graphical user interface (GUI). As the methods and processes described herein modify the data maintained by the non-volatile storage device, thereby transforming the state of the non-volatile storage device, the state of the display subsystem 608 may likewise be transformed to visually represent the change in the underlying data. The display subsystem 608 may include one or more display devices utilizing nearly any type of technology. Such display devices may be combined with the logic processor 602, volatile memory 604, and / or non-volatile storage device 606 in a shared enclosure, or such display devices may be peripheral display devices.

[0063] If included, the input subsystem 610 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and input act translation and / or processing may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition, infrared cameras, color cameras, stereo cameras, and / or depth cameras for machine vision and / or gesture recognition, head trackers, eye trackers, accelerometers and / or gyroscopes for motion detection and / or intent recognition, and electric field sensing components and / or any other suitable sensors for assessing brain activity.

[0064] If included, the communications subsystem 612 may be configured to communicatively couple the various computing devices described herein with each other and with other devices. The communications subsystem 612 may include wireless and / or wireless communication devices compatible with one or more different communications protocols. As a non-limiting example, the communications subsystem may be configured for communication over a wireless telephone network or a wired or wireless local or wide area network (e.g., HDMI over a Wi-Fi connection). In some embodiments, the communications subsystem may enable the computing system 600 to send and / or receive messages to and / or from other devices over a network, such as the Internet.

[0065] The following paragraphs provide additional support for the claims of this application. One aspect provides a computing device. The computing device may include a transformer including an encoder having a global layer and a local layer. The global layer may be configured to receive, for each of a plurality of local input sequences within a global input sequence, a tokenized embedding of each of a plurality of tokens in the local input sequence from the embedding layer. The global layer may be further configured to calculate a global self-attention vector for each of the tokenized embeddings in the local input sequence. The local layer may be configured to receive the global self-attention vector for each local input sequence from the global layer and calculate a local self-attention for the local input sequence. The local layer may be further configured to add the local self-attention vector to the global self-attention vector and normalize it, thereby generating an encoder representation including a self-attention vector for each local input sequence that includes both a global self-attention value and a local self-attention value. The transformer may be configured to output a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence.

[0066] According to this aspect, the global layer may include a state space model layer configured to receive, from the embedding layer, a tokenized embedding of each of the plurality of tokens in the local input sequence.

[0067] According to this aspect, the state-space model layer may include a discrete-time structured state-space sequence model parameterized by a normal matrix plus a low-rank matrix.

[0068] According to this aspect, the discrete-time structured state space sequence model may be an S4 model.

[0069] According to this aspect, computing global self-attention using a global layer including a state space model layer can be achieved with linear computational and memory complexity relative to the global input sequence.

[0070] According to this aspect, the global layer may further include a local layer located in a parallel data path to the state space model layer.

[0071] According to this aspect, the local layer may be configured to receive, from the embedding layer, a tokenized embedding of each of a plurality of tokens in the local input sequence, and to compute a local self-attention for the local input sequence.

[0072] According to this aspect, the global layer may further include a synthesis layer configured to concatenate the global self-attention and the local self-attention calculated in the global layer.

[0073] According to this aspect, the global layer may further include an addition and normalization layer configured to add and normalize the tokenized embeddings output from the embedding layer to the concatenated global and local self-attention computed in the global layer.

[0074] According to this aspect, the global layer may further include a feedforward network configured to receive the normalized combined global and self-attention vectors calculated in the global layer and to output the predicted global layer output at inference time.

[0075] According to this aspect, the transformer may include a classification layer configured to receive the encoder representation and generate a prediction including one or more predicted classifications.

[0076] According to this aspect, the transformer may be a sequence-to-sequence transformer including a decoder including a local layer and a global layer, the decoder configured to receive and decode the encoder representations and generate an output sequence of tokens as a prediction based on a plurality of local input sequences within a global input sequence.

[0077] According to another aspect of the present disclosure, a computerized method is provided.

[0078] According to this aspect, the computerized method may include, for each of a plurality of local input sequences within the global input sequence, receiving, at a global layer of the transformer, from the embedding layer, a tokenized embedding of each of a plurality of tokens in the local input sequence. The computerized method may further include computing, at the global layer, a global self-attention vector for each of the tokenized embeddings in the local input sequence. The computerized method may further include receiving, at the local layer, a global self-attention vector for each local input sequence from the global layer. The computerized method may further include computing, at the local layer, a local self-attention for the local input sequence. The computerized method may further include adding the local self-attention vector to the global self-attention vector and normalizing it to thereby generate an encoder representation including a self-attention vector for each local input sequence, the self-attention vector including both a global self-attention value and a local self-attention value. The computerized method may further include outputting a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence.

[0079] According to this aspect, the global layer may include a state-space model layer, and the computerized method may further include receiving, at the state-space model layer, from the embedding layer, a tokenized embedding of each of the plurality of tokens in the local input sequence.

[0080] According to this aspect, the state-space model layer may include a discrete-time structured state-space sequence model parameterized by a normal matrix plus a low-rank matrix.

[0081] According to this aspect, the global layer may further include a local layer located in a parallel data path to the state space model layer, and the method may further include receiving, at the local layer, a tokenized embedding of each of the plurality of tokens in the local input sequence from the embedding layer, and computing local self-attention for the local input sequence.

[0082] According to this aspect, the global layer may further include a synthesis layer, and the method may further include concatenating the global self-attention and the local self-attention calculated in the global layer in the synthesis layer.

[0083] According to this aspect, the global layer may further include a summing and normalizing layer, and the method may further include adding, in the summing and normalizing layer, the tokenized embeddings output from the embedding layer to the concatenated global and local self-attention calculated in the global layer and normalizing it.

[0084] According to this aspect, the global layer may further include a feedforward network, and the method may further include receiving in the feedforward network the normalized combined global and self-attention vectors calculated in the global layer and output during prediction, and outputting the predicted global layer output from the feedforward network.

[0085] According to another aspect of the present disclosure, a computer device is provided. The computing device may include a transformer including an encoder having a global layer and a local layer. The global layer may be configured to receive, for each of a plurality of local input sequences within a global input sequence, a tokenized embedding of each of a plurality of tokens in the local input sequence from the embedding layer. The global layer may be further configured to calculate a global self-attention vector for each of the tokenized embeddings in the local input sequence. The local layer may be configured to receive the global self-attention vector for each local input sequence from the global layer and calculate a local self-attention for the local input sequence. The local layer may be further configured to add the local self-attention vector to the global self-attention vector and normalize it to thereby generate an encoder representation including a self-attention vector for each local input sequence that includes both a global self-attention value and a local self-attention value. The transformer may be configured to output a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence. The global layer may include a state-space model layer configured to receive, from the embedding layer, a tokenized embedding of each of the plurality of tokens in the local input sequence. The state-space model layer may include a discrete-time structured state-space sequence model parameterized by a normal matrix plus a low-rank matrix. Computing global self-attention using the global layer including the state-space model layer may be achieved with linear computational and memory complexity relative to the global input sequence.

[0086] It is understood that the configurations and / or techniques described herein are exemplary in nature, and that these specific embodiments or examples are not to be construed in a limiting sense, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. Thus, various acts shown and / or described may be performed in parallel, in other sequences, or omitted in the sequence shown and / or described. Similarly, the order of the processes described above may be altered.

[0087] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various processes, systems and configurations and other features, functions, acts and / or properties disclosed herein, and any equivalents thereof.

Claims

1. 1. A computing device including a transformer including an encoder, the encoder comprising: for each of a plurality of local input sequences within a global input sequence, receiving from an embedding layer a tokenized embedding of each of a plurality of tokens in the local input sequence; computing a global self-attention vector for each of the tokenized embeddings in the local input sequence; a global layer configured to receiving the global self-attention vector for each local input sequence from the global layer and calculating local self-attention for the local input sequence; adding the local self-attention vector to the global self-attention vector and normalizing it, thereby generating an encoder representation including a self-attention vector for each local input sequence, the self-attention vector including both a global self-attention value and a local self-attention value; A local layer configured to and The computing device, wherein the transformer is configured to output a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence.

2. 2. The computing device of claim 1, wherein the global layer includes a state space model layer configured to receive, from the embedding layer, the tokenized embedding of each of the plurality of tokens in the local input sequence.

3. The computing device of claim 2 , wherein the state-space model layer comprises a discrete-time structured state-space sequence model parameterized by a normal matrix plus a low-rank matrix.

4. The computing device of claim 3 , wherein the discrete-time structured state space sequence model is an S4 model.

5. 3. The computing device of claim 2, wherein computing the global self-attention using the global layer including the state space model layer is achieved with linear computational complexity and linear memory complexity with respect to the global input sequence.

6. The computing device of claim 2 , wherein the global layer further comprises a local layer located in a parallel data path to the state space model layer.

7. 7. The computing device of claim 6, wherein the local layer is configured to receive from the embedding layer the tokenized embedding of each of the plurality of tokens in the local input sequence and to compute local self-attention for the local input sequence.

8. The computing device of claim 7 , wherein the global layer further comprises a synthesis layer configured to concatenate the global self-attention and the local self-attention calculated in the global layer.

9. 10. The computing device of claim 8, wherein the global layer further comprises an addition and normalization layer configured to add and normalize the tokenized embeddings output from the embedding layer to the concatenated global and local self-attention computed in the global layer.

10. 10. The computing device of claim 9, wherein the global layer further comprises a feedforward network configured to receive the normalized combined global and self-attention vectors calculated in the global layer and to output predicted global layer outputs at inference time.

11. The computing device of claim 10 , wherein the transformer includes a classification layer configured to receive the encoder representation and generate the prediction, the prediction including one or more predicted classifications.

12. 11. The computing device of claim 10, wherein the transformer is a sequence-to-sequence transformer including a decoder including a local layer and a global layer, the decoder configured to receive and decode the encoder representations and generate an output sequence of tokens as the prediction based on the plurality of local input sequences in the global input sequence.

13. receiving, for each of a plurality of local input sequences in a global input sequence, from an embedding layer, at a global layer of a transformer, a tokenized embedding of each of a plurality of tokens in the local input sequence; Computing at the global layer a global self-attention vector for each of the tokenized embeddings in the local input sequence; receiving, at a local layer, the global self-attention vector for each local input sequence from the global layer; Computing local self-attention for the local input sequence at the local layer; adding the local self-attention vector to the global self-attention vector and normalizing it, thereby generating an encoder representation including a self-attention vector for each local input sequence that includes both a global self-attention value and a local self-attention value; outputting a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence; A computerized method comprising:

14. the global layer includes a state space model layer, and the method further comprises: receiving at the state space model layer from the embedding layer the tokenized embedding of each of the plurality of tokens in the local input sequence; 14. The computerized method of claim 13, further comprising:

15. 15. The computerized method of claim 14, wherein the state-space model layer comprises a discrete-time structured state-space sequence model parameterized by a normal matrix plus a low-rank matrix.

16. The global layer further includes a local layer located in a parallel data path to the state space model layer, and the method further comprises: receiving at the local layer from the embedding layer the tokenized embedding of each of the plurality of tokens in the local input sequence, and computing a local self-attention for the local input sequence; 15. The computerized method of claim 14, further comprising:

17. The global layer further comprises a composite layer, and the method further comprises: Concatenating the global self-attention and the local self-attention calculated in the global layer in the synthesis layer.

17. The computerized method of claim 16, further comprising:

18. The global layer further includes a summation and normalization layer, and the method further comprises: In the summation and normalization layer, the tokenized embedding output from the embedding layer is added to the concatenated global and local self-attention computed in the global layer and normalized.

20. The computerized method of claim 17, further comprising:

19. The global layer further includes a feedforward network, and the method further comprises: receiving the normalized combined global and self-attention vectors calculated in the global layer and output during prediction in the feedforward network, and outputting predicted global layer outputs from the feedforward network; 20. The computerized method of claim 18, further comprising:

20. 1. A computing device including a transformer including an encoder, the encoder comprising: for each of a plurality of local input sequences within a global input sequence, receiving from an embedding layer a tokenized embedding of each of a plurality of tokens in the local input sequence; computing a global self-attention vector for each of the tokenized embeddings in the local input sequence; a global layer configured to receiving the global self-attention vector for each local input sequence from the global layer and calculating local self-attention for the local input sequence; adding the local self-attention vector to the global self-attention vector and normalizing it, thereby generating an encoder representation including a self-attention vector for each local input sequence, the self-attention vector including both a global self-attention value and a local self-attention value; A local layer configured to and the transformer is configured to output a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence; the global layer includes a state space model layer configured to receive from the embedding layer the tokenized embedding of each of the plurality of tokens in the local input sequence; the state-space model layer includes a discrete-time structured state-space sequence model parameterized by a normal matrix plus a low-rank matrix; and A computing device, wherein the computation of the global self-attention using the global layer including the state space model layer is achieved with linear computational complexity and linear memory complexity with respect to the global input sequence.

Images