Large language model architecture for high quality and high throughput

Abstract

The embodiments may include a computer readable medium including instructions that when executed by one or more processing devices cause the one or more processing devices to perform a method. The method may include inputting a prompt to a trained model. The trained model may be a Large Language Model comprising a plurality of neural network layers including: at least one first layer comprising an attention module associated with a Transformer deep learning architecture; and at least one second layer comprising a state space module associated with a Mamba deep learning architecture. The method may further include receiving an output of the trained model.

Description

LARGE LANGUAGE MODEL ARCHITECTURE FOR HIGH QUALITY AND HIGH THROUGHPUTRelated Applications

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 564,073, filed on March 12, 2024. The foregoing application is incorporated herein by reference in its entirety.Background

[0002] Individuals and organizations are increasingly employing deep learning models as tools for business as well as personal use. Large Language Models (LLMs), for example, are designed for natural language processing tasks such as language generation, among other tasks. Recent advances in deep learning model architecture have improved the performance of these models, further increasing their prevalence. For example, the transformer deep learning architecture was developed based on a multi -head attention mechanism. The transformer architecture lacks recurrency and thus advantageously requires less training time than earlier recurrent neural architectures (RNNs) such as long short-term memory (LSTM). The transformer architecture has become a widely popular architecture for training LLMs. The Mamba deep learning architecture was developed as an alternative to the transformer architecture for language modeling. The Mamba architecture is based on the Structured State Space sequence model (S4) and introduces improvements in the treatment of time-variant operations. Accordingly, the Mamba architecture advantageously has the increased capability of handling long distance relationships with a simplified architecture.

[0003] These deep learning architectures are not without their drawbacks, however. The transformer architecture is computationally inefficient on long sequences and has a significant memory footprint. While the Mamba architecture is significantly more efficient in processing long sequences compared to transformers, the structured state space models have a significantly lower performance in various language processing tasks.

[0004] Accordingly, in view of these and other deficiencies, there is a significant need for a model architecture providing an improved performance in language processing tasks while maintaining efficiency in long sequences and reducing memory footprint. The disclosed embodiments provide an improved hybrid Transformer-Mamba architecture, that interleaves blocks of Transformer and Mamba layers. The disclosed embodiments thus provide the benefits of both model families. The disclosed embodiments may further include mixture-of-experts (MoE) blocks added in some layers to increase the model capacity while keeping effective parameter usage manageable. The disclosed architecture thus obtains improved performance onstandard language modeling benchmarks relative to previous architectures, while maintaining high throughput and small memory footprint.SUMMARY

[0005] Embodiments consistent with the present disclosure provide an improved deep learning model architecture. In an embodiment, a method for generating an output using a trained model may include inputting a prompt to the trained model. The trained model may be a Large Language Model comprising a plurality of neural network layers, the plurality of neural network layers including: at least one first layer comprising an attention module, wherein the attention module is associated with a Transformer deep learning architecture; and at least one second layer comprising a state space module, wherein the state space module is associated with a Mamba deep learning architecture. The method may further include receiving an output of the trained model.

[0006] In an embodiment, a computer readable medium storing data for access by at least one application program may be executable by one or more processing devices. The non- transitory computer readable medium may comprise a data structure comprising a trained model, the trained model being a Large Language Model comprising a plurality of neural network layers. The plurality of neural network layers may include: at least one first layer comprising an attention module, wherein the attention module is associated with a Transformer deep learning architecture; and at least one second layer comprising a state space module, wherein the state space module is associated with a Mamba deep learning architecture.

[0007] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.BRIEF DESCRIPTION OF DRAWING(S)

[0008] Fig. 1 is a diagram illustrating an exemplary system environment in which the disclosed writing assistant may be used, consistent with disclosed embodiments.

[0009] Fig. 2 illustrates an example model block of the disclosed hybrid Transformer- Mamba mixture -of-experts architecture, consistent with the disclosed embodiments.

[0010] Figs. 3A, 3B, 3C, and 3D are diagrams illustrating example layers that may be included in the disclosed architecture in further detail, consistent with the disclosed embodiments.

[0011] Fig. 4 is a diagram illustrating an example process that may incorporate a model using the disclosed architecture, consistent with the disclosed embodiments.

[0012] Fig. 5 is a flowchart showing an example process for a method, consistent with the disclosed embodiments.DETAILED DESCRIPTION

[0013] The disclosed embodiments relate to a model architecture that combines one or more transformer layers with layers of a recent state-space model, such as a Mamba architecture. The disclosed model architecture thus combines two orthogonal architectural improvements that together give it improved performance and higher throughput, while maintaining a manageable memory footprint.

[0014] The disclosed model architecture (referred to as “Amba”) provides significant improvements over existing deep learning architectures. Despite the immense popularity of the transformer architecture as the predominant architecture for language models, it suffers from two main drawbacks. First, the transformer architecture’s quadratic memory usage, required by its key-value cache, prohibits the processing of long contexts. Second, the transformer architecture’s lack of state entails slow inference and low throughput, since each generated token performs a new computation on the entire context. In contrast, older recurrent neural network (RNN) models, which use infinite context summarized in a hidden state, do not suffer from these limitations. However, traditional RNN models are costly to train and struggle with long distance relationships. Recent state space models (SSMs) like Mamba are more efficient than RNNs and are more capable at handling long distance relationships, but still lag behind the performance of large transformer language models.

[0015] The disclosed architecture may take advantage of both model families, combining transformer and Mamba blocks, as described in further detail below. The disclosed architecture may also provide an improvement over traditional Mamba layers in the form of normalization operations that allow for stable training at a large model scale. In some embodiments, the disclosed architecture further includes mixture -of-experts (MoE) layers, which enable increasing the model capacity (total number of parameters) without increasing compute requirements. The disclosed architecture is also flexible in that different desiderata pertaining to memory usage, throughput, and performance may be achieved. For example, one of these properties may be improved in favor of the others by varying the ratio of Transformer, Mamba, and MoE modules in the architecture or various other parameters. For example, increasing the number of Mamba layers may reduce the key-value cache requirement and accordingly the total memory footprint.

[0016] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings and disclosed herein. The systems and methods are described below in no particular order and can be performed in any order and combination. Additionally, various embodiments of the disclosed writing assistant technology may include some or all of the disclosed features and functionality in any combination.

[0017] Fig. 1 is a schematic diagram of an exemplary system environment in which a trained system using the disclosed model architecture may be employed. For example, system 100 may include a plurality of client devices 110 operated by users 120. System 100 may also include a network 130, server 140, internet resources 150, cloud services 160, and databases 170. The components and arrangement of the components included in system 100 may vary. Thus, system 100 may include any number or any combination of the system environment components shown or may include other components or devices that perform or assist in the performance of the system or method consistent with the disclosed embodiments. The components and arrangements shown in Fig. 1 are not intended to limit the disclosed embodiments, as the components used to implement the disclosed processes and features may vary. Additionally, a trained model incorporating the disclosed model architecture may be implemented on any single component shown (e.g., a single mobile device or single PC included in client devices 110) or may be implemented in a network architecture (e.g., one or more features of the disclosed writing assistant systems and methods being implemented on a server 140, associated with one or more cloud services 160, etc. and having connectivity established with one or more client devices 110 via network 130 (e.g., a WAN, LAN, Internet connection, etc.).

[0018] As shown in Fig. 1, client devices 110 may include a variety of different types of devices, such as personal computers, mobile devices like smartphones and tablets, client terminals, supercomputers, etc. Client devices 110 may include one or more processors for performing various tasks, consistent with the disclosed embodiments. For example, the processor may take the form of, but is not limited to, a microprocessor, embedded processor, or the like, or may be integrated in a system on a chip (SoC). Furthermore, according to some embodiments, the processor may be from the family of processors manufactured by Intel®, AMD®, Qualcomm®, Apple®, NVIDIA®, or the like. The processor may also be a processor based on the ARM architecture, a processor based on the RISC-V architecture, a mobile processor, a graphics processing unit, or any other form of processor. In some embodiments, the processor may be a graphics processing unit (GPU). The disclosed embodiments are not limited to any particular type of processor configured in client devices 110.

[0019] Client devices 110 may further include at least one memory, which may store instructions executed by the processor. Accordingly, client devices 110 may include one or more storage devices configured to store instructions used by the processor to perform functions described herein. The disclosed embodiments are not limited to particular software programs or devices configured to perform dedicated tasks. For example, the memory may store a single program, such as a user-level application, that performs the functions associated with the disclosed embodiments, or may comprise multiple software programs. Additionally, theprocessor may, in some embodiments, execute one or more programs (or portions thereof) remotely located from client devices 110. Furthermore, the memory may include one or more storage devices configured to store data for use by the programs. The memory may include, but is not limited to a hard drive, a solid state drive, a CD-ROM drive, a transient or temporary storage device (e.g., a random-access memory (“RAM”)), a peripheral storage device (e.g., an external hard drive, a USB drive, etc.), a network drive, a cloud storage device, or any other storage device. While described in association with client devices 110, various other components of system 100 may equally be equipped with a processor and memory for performing the disclosed functions.

[0020] Client devices 110 may be connected to a network such as network 130. In some cases, a user 120 may access an application that relies, at least in part, on a trained system using the disclosed model architecture. For example, the trained system may be incorporated into a dedicated application on a client device 110, or may be incorporated into other forms of applications including, but not limited to, email editors, word processing programs, presentation applications, spreadsheet applications, PDF editors, etc.

[0021] Network 130, in some embodiments, may comprise one or more interconnected wired or wireless data networks that receive data from one device (e.g., client devices 110) and send it to another device (e.g., one or more servers 140). For example, network 130 may be implemented to include one or more Internet communication paths, a wired Wide Area Network (WAN), a wired Local Area Network (LAN), a wireless LAN (e.g., Bluetooth®, etc.), or the like. Each component in system 100 may communicate bidirectionally with other system 100 components either through network 130 or through one or more direct communication links (not shown).

[0022] As described, a model incorporating the disclosed architecture may be implemented and run using a variety of different equipment, such as one or more servers, personal computers, mobile devices, supercomputers, mainframes, or the like, connected via various types of networks. In some embodiments, the trained model may be configured to receive information from client device 110, database 170, server 140, cloud service 160, and / or Internet sources 150 (among others) and send or return information to the same. The trained model can be incorporated into client devices 110 and run locally or be run on a server 140 or from a cloud service 160 accessed by the client device 110 via network 130.

[0023] As indicated above, the disclosed model architecture may be a hybrid decoder architecture that mixes attention layer blocks with state-space model (SSM) model blocks. For example, the attention layers may correspond to a transformer deep learning architecture, which may be based on a multi -head attention mechanism. In the transformer architecture, text may beconverted to numerical tokens, and each token may be converted into a vector via a word embedding table. The transformer architecture may have no recurrent units and therefore may allow for a relatively fast training time.

[0024] The SSM model blocks, on the other hand, may be based on recurrent models that selectively process information based on the current input. Accordingly, the SSM blocks may focus on relevant information and discard irrelevant data. For example, the SSM blocks may be Mamba layer blocks, which may replace the complex attention and MLP blocks of Transformers with a single, unified SSM block. The disclosed architecture may also include mixture-of-experts (MoE) blocks in which multiple expert networks may be used to divide a problem space into homogeneous regions. The MoE blocks may increase the model capacity while keeping effective parameter usage manageable.

[0025] Fig. 2 illustrates an example model block 200 of the disclosed hybrid Transformer-Mamba mixture-of-experts architecture, consistent with the disclosed embodiments. As shown in Fig. 2, each block 200 may include a sequence of layers. For example, block 200 may include one or more transformer layers 210 which may be attention layers, as explained above. Block 200 may further include one or more Mamba layers 220, which may be improved SSM layers. In some embodiments, a MoE module may be applied to various layers, as indicated above. For example, block 200 may include one or more Mamba layers 230 including MoE modules. As discussed in further detail below, the particular arrangement and number of layers 210, 220, and 230 shown in Fig. 2 are provided by way of example, and various other arrangements may be implemented.

[0026] Consistent with the disclosed embodiments, each layer may include one or more modules. For example, each layer may include either an attention module or a SSM module. Figs. 3A, 3B, 3C, and 3D are diagrams illustrating example layers that may be included in the disclosed architecture in further detail, consistent with the disclosed embodiments. As indicated above, block 200 may include one or more transformer layers 210, which may include an attention module along with various other modules. Fig. 3A illustrates an example transformer layer 310, consistent with the disclosed embodiments. Transformer layer 310 may include an attention module 316, which may apply a self-attention mechanism, consistent with a transformer deep learning architecture. Attention module 316 may be followed by a multilayer perceptron (MLP) module 312. Multilayer perceptron module 312 may consist of fully connected neurons and may include nonlinear activation functions. In some embodiments, attention module 316 and multilayer perceptron module 312 may be preceded by normalization functions 314 and 318, respectively. Normalization functions 314 and 318 may be mathematical operations applied to data within layer 310 that rescale the rescale the activation of hiddenneurons within the neural network. For example, in some embodiments, normalization functions 314 and 318 may be Root mean square layer normalization (RMSNorm) functions. Various other normalization functions may be used, such as LayerNorm, Adaptive layer norm (adaLN), or various other activation normalization functions.

[0027] In some embodiments, block 200 may include one or more attention layers having a mixture -of-experts (MoE) modules applied. Fig. 3B illustrates example attention layer 320 having a mixture-of-experts (MoE) module applied, consistent with the disclosed embodiments. Attention layer 320 may include an attention module 326, similar to transformer layer 310. Accordingly, attention module 326 may be the same as or similar to attention module 316 described above. However, instead of a multilayer perceptron module, such as multilayer perceptron module 312, attention layer 320 may include mixture-of-experts module 322, as shown in Fig. 3B. Mixture-of-experts module 322 may increase efficiency of the neural network by routing inputs to experts that are most effective for processing the input.

[0028] As indicated above, block 200 may further include one or more Mamba layers, which may include a Mamba module along with various other modules. Fig. 3C illustrates an example Mamba layer 330, consistent with the disclosed embodiments. Mamba layer 330 may include a Mamba module 336, which may include selection mechanism that adapts structured state space model (SSM) parameters based on the input, consistent with a Mamba deep learning architecture. Mamba module 336 may be followed by a multilayer perceptron module 332, which may be the same as or similar to multilayer perceptron module 312. Similar to transformer layer 310, attention module 336 and multilayer perceptron module 332 may be preceded by normalization functions 334 and 338, respectively, which may be the same as or similar to normalization functions 314 and 318.

[0029] In some embodiments, block 200 may include one or more Mamba layers having a mixture-of-experts (MoE) modules applied, as indicated above. Fig. 3C illustrates example Mamba layer 340 having a mixture-of-experts module, consistent with the disclosed embodiments. Mamba layer 340 may include a Mamba module 346, similar to Mamba layer 330. However, instead of a multilayer perceptron module, Mamba layer 340 may include mixture-of-experts module 342, as shown in Fig. 3D. Mixture-of-experts module 322 may increase efficiency of the neural network by routing inputs to experts that are most effective for processing the input.

[0030] While block 200 is shown with a particular arrangement and number of layers, it is to be understood that this arrangement is provided by way of example. The disclosed model architecture may include any arrangement of layers consisting of at least one attention layer and at least one state-space model layer. The particular arrangement of layers may vary dependingon the particular implementation. Accordingly, the disclosed model architecture may allow for different combinations of layers or other variations, each of which may provide different advantages. For example, block 200 may contain I layers, including a number of attention layers, a, and a number of Mamba layers, m. Accordingly, block 200 may include layers mixed at a ratio of a: m, meaning a attention layers for every m Mamba layers. In the example shown in Fig. 2, the ratio a: m may be 1 :7 (or, conversely, a ratio m: a of 7: 1). As indicated above, some of Mamba layers may include MoE modules (e.g., Mamba layer 230). A value e may represent how often a MoE module (e.g., MoE module 342) is used instead of a single MLP module. In other words, the MoE module may be applied every e layers. In the example shown in Fig. 2, the MoE module may be applied every other Mamba layer (i.e., e = 2). Further, the MoE module may include a router for selecting the most effective experts for the input. Various parameters for the MoE module may also be customized. For example, the number of possible experts per layer may be defined by a value, n. The MoE module may be configured to choose the top K experts at each token. As one example, MoE module 342 may be configured with 16 total experts and may use 2 of these experts at each token. This configuration is provided by way of example, however, and the values, n and K may be customized to vary performance attributes of the model.

[0031] Accordingly, the disclosed model architecture may be customized according to at least the following degrees of freedom:• / : The number of layers;• a. fraction of attention layers;• m fraction of Mamba layers;• n: total number of experts per layer;• K: number of top experts used at each token; and• e: how often to use MoE instead of a single MLP.

[0032] Given this design space, the disclosed model architecture may provide flexibility in promoting certain properties or characteristics over others. For example, increasing m and decreasing a (i.e., increasing the fraction of Mamba layers at the expense of attention layers) may reduce the required memory for storing the key-value cache. This may also reduce the overall memory footprint, which may be beneficial for processing long sequences. Increasing the fraction of Mamba layers may also improves throughput, especially at long sequences. Additionally, balancing n, K, and e may affect the relationship between effective parameters and total parameters. Thus, by increasing n the model capacity may be increased while maintaining a smaller latency. Similarly, by increasing K, the compute of the model may be increased whilemaintaining the same model capacity. Accordingly, the particular parameters of block 200 may vary depending on the particular application.

[0033] Through the combination of the various layers described herein, the disclosed model architecture may provide improved performance on standard language modeling benchmarks while maintaining high throughput and small memory footprint. For example, the disclosed architecture may excel at long context setups and may be extremely efficient in terms of latency and throughput. Moreover, the disclosed model architecture may not use any positional information in the form of positional embeddings, changes to the attention weights, and so on. Through the use of (sequential) Mamba layers before the attention layer, adding explicit positional information is not necessary. Given these advantages, a model using the disclosed architecture may be applied in a wide variety of applications.

[0034] Fig. 4 illustrates an example process 400 that may incorporate a model using the disclosed architecture, consistent with the disclosed embodiments. For example, process 400 may include a trained model 410, which may include one or more attention (e.g., transformer) layers and one or more Mamba layers, consistent with block 200 described above. For example, trained model 410 may be a large language model (LLM) or other form of deep learning model. As shown in Fig. 4, process 400 may include receiving a user input 420. For example, user input 420 may be provided by user 120 using the various client devices 110 described above. Based on user input 420, a prompt 422 may be generated for input to trained model 410. In some embodiments, user input 420 may be formatted such that it may be input to trained model 410 directly. Accordingly, in such embodiments, prompt 422 may correspond to user input 420. Alternatively, or additionally, the disclosed system may be configured to generate prompt 422. For example, this may include some or all of user input 422 to a standard format, such as a CSV format, a JSON format, or the like. Trained model 410 may then generate a model output 430, which may be converted to a response 432 to be presented in response to user input 420. In some embodiments, generating response 432 may include converting model output 430 to a format more accessible to user 120. For example, model output 430 may be presented in a standardized format (e.g., a JSON format), which may be converted to a natural language format in response 432.

[0035] In some embodiments, some or all of process 400 may be performed as part of a writing assistant application. For example, the disclosed model architecture may be used in a writing assistant program configured to assist users with natural language processing tasks. Accordingly, user input 420 may be a text-based input, such as at least one word conveying an idea. Model output 430 (and / or response 432) may be a textual output option that differs in some way from user input 420. For example, the response may present the idea conveyed in userinput 420 in a different tone, using different vocabulary, using a fewer or greater number of words, or the like. While a writing assistant application is provided by way of example, the disclosed model architecture may be implemented in a wide variety of applications, and the various parameters defined above may be varied depending on the application.

[0036] Fig. 5 is a flowchart showing an example process 500, consistent with the disclosed embodiments. Process 500 may be performed by at least one processor of a computing device, such as a processor of computing device 110, described above. It is to be understood that throughout the present disclosure, the term “processor” is used as a shorthand for “at least one processor.” In other words, a processor may include one or more structures that perform logic operations whether such structures are collocated, connected, or dispersed. In some embodiments, a non-transitory computer readable medium may contain instructions that when executed by a processor cause the processor to perform process 500. Further, process 500 is not necessarily limited to the steps shown in Fig. 5, and any steps or processes of the various embodiments described throughout the present disclosure may also be included in process 500, including those described above with respect to, for example, Fig. 4.

[0037] In step 510, process 500 includes inputting a prompt to a trained model. For example, step 510 may include inputting prompt 422 into trained model 410, as described above. The trained model may be based on the model architecture disclosed herein. For example, the trained model may be a Large Language Model comprising a plurality of neural network layers, as described above with respect to block 200. Consistent with the disclosed embodiments, the plurality of neural network layers may include at least one first layer comprising an attention module. For example, the trained model may include one or more first layers corresponding to transformer layer 210, which may include attention module 316 (or attention module 326). The attention module may be associated with a Transformer deep learning architecture, as described above.

[0038] The plurality of neural network layers may further include at least one second layer comprising a state space module. The state space module may be associated with a Mamba deep learning architecture, as described above. For example, the trained model may include one or more second layers corresponding to Mamba layers 220 and / or 230, which may include Mamba module 336 or 346, respectively.

[0039] Consistent with the disclosed embodiments, the layers may be configured in a variety of different arrangements. For example, a ratio of the number of a number of second layers comprising state space modules to first layers comprising attention modules may be varied depending on the particular application. As one example, a ratio of the number of a number ofsecond layers comprising state space modules to first layers comprising attention modules may be at least 7: 1.

[0040] In some embodiments, the plurality of layers may further include various modules in addition to the attention modules and state space modules. For example, the at least one first layer may comprise a plurality of first layers. A first portion of the plurality of first layers may further comprise a multi-layer perceptron (MLP) module. For example, the first portion of the plurality of first layers may correspond to transformer layer 310 and may include multilayer perceptron module 312. In some embodiments, a second portion of the plurality of first layers may further comprise a mixture of experts (MoE) module. For example, the second portion of the plurality of first layers may correspond to attention layer 320 and may include mixture-of-experts module 322.

[0041] Similarly, the at least one second layer may comprise a plurality of second layers. A first portion of the plurality of second layers may further comprise a multi-layer perceptron (MLP) module. For example, the first portion of the plurality of second layers may correspond to Mamba layer 330 and may include multilayer perceptron module 332. In some embodiments, a second portion of the plurality of second layers may further comprise a mixture of experts (MoE) module. For example, the second portion of the plurality of second layers may correspond to Mamba layer 340 and may include mixture-of-experts module 342.

[0042] In some embodiments, each of the plurality of layers may include at least one normalization module. For example, the plurality of first layers may further comprise one or more normalization modules 314, 318, 324, or 328 and the plurality of second layers may further comprise one or more normalization modules 334, 338, 344, or 348. In some embodiments, the normalization modules may include Root Mean Square Normalization (RMSNorm) modules, as described above.

[0043] In step 520, process 500 includes receiving an output of the trained model. For example, step 520 may include receiving model output 430, as described above. Consistent with the disclosed embodiments, the trained model may generate the output without using a positional encoding. In other words, the inclusion of the at least one first layer comprising an attention module and the at least one second layer comprising a state space module may negate any need for positional encodings and thus the plurality of layers may not include positional encodings.

[0044] In some embodiments, the output of the trained model may be output to or shown on a display of a device (e.g., a display associated with a computer, including a handheld device such as a phone, a watch, or a tablet). In yet other embodiments, the output of the trained model may be stored (e.g., locally) and / or transmitted over one or more networks (e.g., network 130) to an external storage location.

[0045] The systems and methods described above are presented in no particular order and can performed in any order and combination. For example, various embodiments of the writing assistant may include a combination of all of the features and functionality described above, or in some cases, the writing assistant may offer any subset of described features and / or functionality.

[0046] The above-described systems and method can be executed by computer program instructions that may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce instructions which when implemented cause the writing assistant to perform the above-described methods.

[0047] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the above-described methods.

[0048] It will be understood from the foregoing description that modifications and changes may be made in various embodiments of the present invention without departing from the invention described in this specification. The descriptions in this specification are for purposes of illustration only and are not to be construed in a limiting sense. The scope of the present invention is limited only by the language of the following claims.

Claims

CLAIMSWhat is claimed is:

1. A non-transitory computer readable medium including instructions that when executed by one or more processing devices cause the one or more processing devices to perform a method including: inputting a prompt to a trained model, wherein the trained model is a Large Language Model comprising a plurality of neural network layers, the plurality of neural network layers including: at least one first layer comprising an attention module, wherein the attention module is associated with a Transformer deep learning architecture; and at least one second layer comprising a state space module, wherein the state space module is associated with a Mamba deep learning architecture; and receiving an output of the trained model.

2. The non-transitory computer readable medium of claim 1, wherein each of the plurality of layers includes at least one normalization module.

3. The non-transitory computer readable medium of claim 1, wherein the at least one second layer comprises a plurality of second layers and wherein a first portion of the plurality of second layers further comprise a multi-layer perceptron (MLP) module.

4. The non-transitory computer readable medium of claim 1, wherein the at least one second layer comprises a plurality of second layers and wherein a second portion of the plurality of second layers further comprise a mixture of experts (MoE) module.

5. The non-transitory computer readable medium of claim 1, wherein a ratio of the number of a number of second layers comprising state space modules to first layers comprising attention modules is at least 7: 1.

6. The non-transitory computer readable medium of claim 1, wherein the at least one first layer comprises a plurality of first layers and wherein a first portion of the plurality of first layers further comprise a multi-layer perceptron (MLP) module.

7. The non-transitory computer readable medium of claim 1, wherein the at least one first layer comprises a plurality of first layers and wherein a second portion of the plurality of first layers further comprise a mixture of experts (MoE) module.

8. The non-transitory computer readable medium of claim 1, wherein the trained model generates the output without using a positional encoding.

9. The non-transitory computer readable medium of claim 1, wherein the method further includes: receiving a user input via a user interface; generating the prompt based on the user input; and providing a response to the user input via the user interface, the response being based on the output of the trained model.

10. A computer-implemented method for generating an output using a trained model, the method including: inputting a prompt to a trained model, wherein the trained model is a Large Language Model comprising a plurality of neural network layers, the plurality of neural network layers including: at least one first layer comprising an attention module, wherein the attention module is associated with a Transformer deep learning architecture; and at least one second layer comprising a state space module, wherein the state space module is associated with a Mamba deep learning architecture; and receiving an output of the trained model.

11. The method of claim 10, wherein each of the plurality of layers includes at least one normalization module.

12. The method of claim 10, wherein the at least one second layer comprises a plurality of second layers and wherein a first portion of the plurality of second layers further comprise a multi-layer perceptron (MLP) module.

13. The method of claim 10, wherein the at least one second layer comprises a plurality of second layers and wherein a second portion of the plurality of second layers further comprise a mixture of experts (MoE) module.

14. The method of claim 10, wherein a ratio of the number of a number of second layers comprising state space modules to first layers comprising attention modules is at least 7: 1.

15. The method of claim 10, wherein the at least one first layer comprises a plurality of first layers and wherein a first portion of the plurality of first layers further comprise a multi-layer perceptron (MLP) module.

16. The method of claim 10, wherein the at least one first layer comprises a plurality of first layers and wherein a second portion of the plurality of first layers further comprise a mixture of experts (MoE) module.

17. The method of claim 10, wherein the trained model generates the output without using a positional encoding.

18. The method of claim 10, further comprising: receiving a user input via a user interface; generating the prompt based on the user input; and providing a response to the user input via the user interface, the response being based on the output of the trained model.

19. A non-transitory computer readable medium storing data for access by at least one application program executable by one or more processing devices, the non-transitory computer readable medium comprising: a data structure comprising a trained model, the trained model being a Large Language Model comprising a plurality of neural network layers, wherein the plurality of neural network layers include: at least one first layer comprising an attention module, wherein the attention module is associated with a Transformer deep learning architecture; andat least one second layer comprising a state space module, wherein the state space module is associated with a Mamba deep learning architecture.

20. The non-transitory computer readable medium of claim 19, wherein the trained model does not include a positional encoding.

Images