Language modeling using factorized memory

Abstract

This invention provides language modeling using factorized memory. [Solution] Language modeling using factorized memory is performed by calculating a topic affinity score for each topic vector based on input token embeddings and a topic affinity weight matrix, updating each topic vector based on the corresponding topic affinity score, and merging the updated topic vectors to generate output token embeddings.

Description

[Technical Field]

[0001] <Cross-reference of related applications> This application claims priority to U.S. Provisional Patent Application No. 63 / 730,898, filed on 11 December 2024, the entire contents of which are incorporated herein by reference.

[0002] This disclosure relates to language modeling using factorized memory. [Background technology]

[0003] The information disclosed in this background section is intended solely to enhance the understanding of the general background of this disclosure and should not be construed as an endorsement or any suggestion of forming prior art already known to those skilled in the art.

[0004] The transformer architecture of a Large-Scale Language Model (LLM) uses a context window to consider the previous L tokens when generating the next token. To generate a sentence of L tokens, the time required is O(L). 2 The calculation of ) is required. [Overview of the project] [Means for solving the problem]

[0005] In at least some embodiments, language modeling using factorized memory is performed by a method of operation that includes: calculating a topic affinity score for each topic vector in a plurality of topic vectors based on input token embeddings and a topic affinity weight matrix; updating each topic vector in at least some of the plurality of topic vectors based on the corresponding topic affinity score; and merging at least some of the updated topic vectors in the plurality of topic vectors to generate output token embeddings.

[0006] In at least some embodiments, language modeling using factorized memory is performed by a device configured to perform operations including: calculating a topic affinity score for each topic vector in a plurality of topic vectors based on input token embeddings and a topic affinity weight matrix; updating each topic vector in at least some of the plurality of topic vectors based on the corresponding topic affinity score; and merging at least some of the updated topic vectors in the plurality of topic vectors to generate output token embeddings.

[0007] In at least some embodiments, language modeling using factorized memory is performed by a non-temporary computer-readable medium that includes instructions causing one or more processors to perform operations including: calculating a topic affinity score for each topic vector in a plurality of topic vectors based on an input token embedding and a topic affinity weight matrix; updating each topic vector in at least some of the plurality of topic vectors based on the corresponding topic affinity score; and merging at least some of the updated topic vectors to generate an output token embedding.

[0008] Features, aspects, and advantages of embodiments of this disclosure are described below with reference to the accompanying drawings, in which similar reference numerals indicate similar elements. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of a factorized memory block of a language model according to at least some embodiments of the present disclosure. [Figure 2] This is an operational flow for utilizing recursive memory states according to at least some embodiments of the present disclosure. [Figure 3] This is an operational flow for updating a topic vector according to at least some embodiments of the present disclosure. [Figure 4] This is an operational flow for merging updated topic vectors according to at least some embodiments of the present disclosure. [Figure 5] This is a schematic diagram of a language model having factorized memory according to at least some embodiments of the present disclosure. [Figure 6] This is an operational flow for building and training a language model using factorized memory, according to at least some embodiments of the present disclosure. [Figure 7] This figure shows one embodiment of a device for language modeling using factorized memory, according to at least some embodiments of the present disclosure. [Modes for carrying out the invention]

[0010] The following disclosure provides many different embodiments or examples for implementing different features of this disclosure. To simplify this disclosure, specific examples of components, values, behaviors, materials, and configurations are described below. Of course, these are merely examples and are not intended to be limiting. Other components, values, behaviors, materials, and configurations are also possible. In addition, this disclosure may repeat reference numbers and / or letters in various examples. This repetition is for simplification and clarity and does not in itself presuppose any relationships between the various embodiments and / or configurations described.

[0011] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, software, or combinations of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not limited to their implementation forms. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code. It will be understood that software and hardware may be designed to implement the systems and / or methods based on the descriptions herein.

[0012] While specific combinations of features are described in the claims and / or disclosed herein, no particular combination is intended to limit the disclosure of implementations. In fact, many of these features may be combined in ways not specifically described in the claims and / or disclosed herein. Even if a dependent claim depends directly on only one claim, this disclosure may indicate that the dependent claim depends on other claims within the set of claims.

[0013] Any elements, actions, or instructions used herein should not be construed as important or essential unless expressly stated otherwise. Furthermore, where used herein, the articles “a” and “an” (in other words, nouns not mentioned in the plural) are intended to include one or more items and may be used interchangeably with “one or more.” Also, where used herein, terms such as “has,” “have,” “having,” “include,” and “including” are intended to be non-restrictive. Additionally, the phrase “based on” is intended to mean “based at least partially” unless otherwise specified. Furthermore, expressions such as “at least one of [A] and [B],” “[A] and / or [B],” or “at least one of [A] or [B]” should be understood to include A only, B only, or both A and B.

[0014] In this disclosure, certain tasks may be performed using AI / ML (Artificial Intelligence / Machine Learning) models. An AI / ML model is a model generated using one or more AI techniques, one or more ML algorithms, or both, which generates output data based on input data. This output data is used to perform a task. Tasks performed using AI / ML models include those generally referred to as intelligent tasks, such as classification, prediction, and natural language processing.

[0015] Although AI and ML are described separately, ML is a technology included in AI. In ML, rather than being explicitly programmed for a specific task, a system can improve its performance over time by identifying patterns from training data and making inferences. Typically, the generation of an ML model involves data collection, model training, and model inference. Data collection includes the collection and preprocessing of data used for training and inference. Model training includes developing and validating a model using the collected data. Model inference includes applying the trained model to new data to generate new output data and performing a task.

[0016] Machine learning includes various types of learning methods such as supervised learning, unsupervised learning, reinforcement learning, semi-supervised learning, self-supervised learning, transductive learning, transfer learning, meta-learning, etc. These types of learning methods may be appropriately selected according to the embodiments. Unless otherwise specified, applications of types not mentioned in this specification are not excluded. In addition, the structure of the ML model may vary depending on the embodiments and learning methods and is not limited to the disclosed methods. Furthermore, ML includes deep learning that uses models including neural networks. Deep learning models can include, for example, deep neural networks (DNNs), convolutional neural networks (CNNs), etc.

[0017] Note that the AI / ML models shown below are just examples and are not limited to the illustrated AI / ML models. These models can be modified or changed by using different AI or ML algorithms. Note that the configuration of the neural network is not limited to the configuration disclosed in this disclosure and can be changed.

[0018] Scaling L to a very large number incurs a very high computational cost. For example, a 1GB email contains hundreds of millions of tokens, far exceeding the limits of commercial APIs (usually 32k - 100k). Transformers do not learn during inference. If you want to adjust their behavior, you need to transmit prompts shorter than L for each conversation turn. The error rate increases with complexity. Prompt engineering and RAG become increasingly error-prone as the task complexity increases.

[0019] The language model according to at least some embodiments of the present disclosure utilizes a recursive memory state that includes an encoded state from previous inputs that form the basis of the output. In at least some embodiments, the recursive memory state is of a fixed memory size.

[0020] In at least some embodiments, a language model that generates an output based on a recursive memory state representing previous inputs requires less memory than a language model that generates an output based on a Transformer architecture that directly considers previous inputs within a context window.

[0021] In at least some embodiments, the hyperspace of token embeddings is partitioned into M fixed topics. In at least some embodiments, each topic centroid

Number

Number

[0022] In at least some embodiments, when an input embedding x t is given, the topic affinity α tThis is calculated across all topic centroids. In at least some embodiments, the final output embedding y t It is formed as a weighted average of memory vectors, α t Use this as a weight.

number

number

number

[0023] In at least some embodiments, memory updates are also gated by topic affinity. In at least some embodiments, only memory vectors corresponding to topics closely related to the input receive large updates, while other topic-specific memories remain unaffected.

number

[0024] In at least some embodiments, negative terms in the original memory update rule can be simplified by taking advantage of the assumption that token embeddings are evenly distributed across their entire topic partition in a well-trained embedding space. In at least some embodiments, input embeddings are RMS or layer-normalized per transformer.

[0025] In at least some embodiments, the scaling factor for updates is defined as follows:

number

[0026] In at least some embodiments,

number

number

number

[0027] or

number

[0028] The model is as follows:

number

[0029] In the aforementioned memory update formula, the sequence can be initialized as a zero tensor at the start of the sequence.

number

[0030] In at least some embodiments, the above can be summarized as a set of expressions described below for utilizing recursive memory states. In at least some embodiments, this set of expressions for utilizing recursive memory states enables scaling to a large number of topic partitions m. In at least some embodiments, α t This acts as a routing probability, skewing updates towards the most relevant partition.

[0031] In at least some embodiments, the following will be described

number

number

[0032] In at least some embodiments, the gating network also uses memory blocks.

number

number

number

number

number

[0033] In at least some embodiments, the network is "sparsely activated," and as a result,

number

[0034] Figure 1 is a schematic diagram of a factorized memory block 100 of a language model according to at least some embodiments of the present disclosure. The factorized memory block 100 includes an input token embedding 101, a memory update function 102, topic vectors 104A, 104B, and 104M, a memory merge weight 106, a memory merge function 108, and an output token embedding 109. In at least some embodiments, the factorized memory block 100 is configured to selectively update a portion of the recursive memory state that forms the basis of the output.

[0035] The input token embedding 101 is an instance of the input to the factorized memory block 100. In at least some embodiments, the input token embedding 101 is configured to represent the tokens of the natural language prompt as vectors in the feature space.

[0036] The memory update function 102 is an element of the factorized memory block 100. In at least some embodiments, the memory update function 102 is configured to update topic vectors such as topic vectors 104A, 104B, and 104M based on topic update weight values. In at least some embodiments, the memory update function 102 is further configured to store the updated topic vectors in physical memory such as memory 763 in Figure 7, which is described below.

[0037] Topic vectors such as topic vectors 104A, 104B, and 104M are elements of the factorized memory block 100. In at least some embodiments, the topic vectors form a recursive memory state. In at least some embodiments, each of the topic vectors 104A, 104B, and 104M is updated by the memory update function 102 based on topic update weight values. In at least some embodiments, only some topic vectors are updated in response to each input token embedding. In at least some embodiments, the topic vectors are merged by the memory merge function 108 to produce an output token embedding 109.

[0038] The memory merge weights 106 are elements of the factorized memory block 100. In at least some embodiments, the memory merge weights 106 are configured to control the merging of topic vectors such as topic vectors 104A, 104B, and 104M based on the affinity of the input token embeddings to each topic. In at least some embodiments, the memory merge weights 106 are configured to skew the influence on the output token embeddings 109 towards topics with higher affinity for the token embeddings. In at least some embodiments, the memory merge weights 106 are calculated using a topic merge rate and a topic affinity score.

[0039] The memory merge function 108 is an element of the factorized memory block 100. In at least some embodiments, the memory merge function 108 is configured to merge updated topic vectors such as topic vectors 104A, 104B, and 104M to generate an output token embedding 109. In at least some embodiments, the memory merge function 108 is configured to compute the output projection of the merged topic vectors. In at least some embodiments, the memory merge function 108 is further configured to read the updated topic vectors from physical memory such as memory 763 in Figure 7, which will be described later.

[0040] The output token embedding 109 is an instance of the output from the factorized memory block 100. In at least some embodiments, the output token embedding 109 is generated by merging updated topic vectors such as topic vectors 104A, 104B, and 104M using the memory merge function 108. In at least some embodiments, the output token embedding 109 is the output projection of the merged topic vectors.

[0041] Figure 2 shows an operational flow for utilizing a recursive memory state according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method for utilizing a recursive memory state. In at least some embodiments, this method is performed by a processor of the device, such as the processor 762 of the device 760 shown in Figure 7, which will be described later.

[0042] In S220, the processor calculates the topic affinity score. In at least some embodiments, the processor receives input token embeddings. In at least some embodiments, the processor normalizes the input token embeddings before calculating the topic affinity score. In at least some embodiments, the normalization is root mean square (RMS) normalization. In at least some embodiments, the processor applies softmax to calculate the topic affinity score. In at least some embodiments, the processor calculates the topic affinity score according to the following formula

number

number

number

number

number

number

number

number

[0043] In at least some embodiments, the processor performs sparse updates. In at least some embodiments, the processor calculates topic affinity scores such that only topics with the highest affinity scores are updated. In at least some embodiments, according to Equation 1 or Equation 2

number

number

number

number

number

number

number

[0044] In S224, the processor updates the topic vectors based on topic update weight values. In at least some embodiments, the processor calculates the updated topic vectors based on the input token embeddings, topic update weights, and previous topic vectors. In at least some embodiments, the processor calculates updated topic vectors for only some topics. In at least some embodiments, the processor updates each topic vector among at least some of a plurality of topic vectors, where the corresponding topic affinity weight value is within a predetermined number of maximum topic affinity weight values. In at least some embodiments, the processor performs sparse updates,

number

number

[0045] In S228, the processor merges the updated topic vectors. In at least some embodiments, the processor calculates the output projection of the merged topic vectors based on the topic merge weights, the updated topic vectors, and the output projection weight values. In at least some embodiments, the processor retrieves the updated topic vectors from physical memory. In at least some embodiments, the processor merges multiple updated topic vectors to generate output token embeddings. In at least some embodiments, the processor merges each topic vector from at least some of multiple topic vectors, where the corresponding topic affinity weight value is within a predetermined number of maximum topic affinity weight values. In at least some embodiments, the processor performs a sparse merge.

number

number

[0046] Figure 3 shows an operation flow for updating a topic vector according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method for updating a topic vector. In at least some embodiments, this method is performed by a processor of the device, such as the processor 762 of the device 760 shown in Figure 7, which will be described later.

[0047] In S330, the processor calculates the topic update rate. In at least some embodiments, the processor calculates the topic update rate using topic update rate weights and input token embeddings. In at least some embodiments, the processor calculates the topic update rate according to the following formula

number

number

number

number

[0048] In S332, the processor calculates a topic update weight. In at least some embodiments, the processor calculates the topic update weight using a topic update rate and a topic affinity score. In at least some embodiments, the processor calculates the topic update weight according to the following formula

Number

Number

[0049] In S334, the processor calculates an input projection. In at least some embodiments, the processor calculates the input projection using an input projection weight matrix and an input token embedding. In at least some embodiments, the processor calculates the input projection according to the following formula

Number

Number

[0050] In S336, the processor calculates an updated topic vector. In at least some embodiments, the processor calculates the updated topic vector using the input projection, the topic update weight, and the previous topic vector. In at least some embodiments, each topic vector among the plurality of topic vectors has a fixed length. In at least some embodiments, the processor calculates the updated topic vector h according to the following formula tCalculate.

number

[0051] In S338, the processor stores the updated topic vectors in memory. In at least some embodiments, the processor stores each updated topic vector in one or more memory banks having a capacity equal to the updated topic vectors. In at least some embodiments, the processor stores the updated topic vectors in memory by overwriting the previous topic vectors. In at least some embodiments, the processor stores the updated topic vectors in memory by saving the previous topic vectors during language model training.

[0052] Figure 4 shows an operation flow for merging updated topic vectors according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method for merging updated topic vectors. In at least some embodiments, this method is performed by a processor of the device, such as the processor 762 of the device 760 shown in Figure 7, which will be described later.

[0053] In S440, the processor calculates the topic merge rate. In at least some embodiments, the processor calculates the topic merge rate using topic merge rate weights and input token embeddings. In at least some embodiments, the processor calculates the topic merge rate according to the following formula

number

number

number

[0054] In S444, the processor calculates the topic merge weights. In at least some embodiments, the processor calculates the topic merge weights using the topic merge rate and the topic affinity score. In at least some embodiments, the processor calculates the topic merge weights according to the following formula

number

number

[0055] In S448, the processor calculates the output projection of the merged topic vector. In at least some embodiments, the processor calculates the output projection using the topic merge weights, the updated topic vector, and the output projection weight values. In at least some embodiments, the processor calculates the output projection according to the following formula

number

number

[0056] Figure 5 is a schematic diagram of a language model having factorized memory according to at least some embodiments of the present disclosure. The language model 510 includes a token embedding layer 512, one or more decoding layers such as a decoder layer 514, and a language model head layer 518. In at least some embodiments, the language model 510 is configured to receive a natural language prompt 511 as input. In at least some embodiments, the language model 510 is configured to produce a natural language response 519 as output. Although the language model 510 is primarily designed for natural language, the natural language prompt 511 and natural language response 519 are not strictly limited to natural language. The natural language prompt 511 and natural language response 519 may include non-verbal text, such as code, mathematical algorithms, programming languages ​​or markup languages, or any other non-verbal elements commonly associated with natural language.

[0057] The token embedding layer 512 is a group of layers included in the language model 510. In at least some embodiments, the token embedding layer 512 is configured to parse the natural language prompt 511 and convert it into tokens. In at least some embodiments, the token embedding layer 512 is configured to embed tokens in a vector in feature space. In at least some embodiments, the token embedding layer 512 is configured to encode the natural language prompt 511 and convert it into an input token embedding, such as the input token embedding 101 in Figure 1. In at least some embodiments, the token embedding layer 512 is generally compatible with the language model. In at least some embodiments, the token embedding layer 512 can be trained independently of the language model 510. In at least some embodiments, the token embedding layer 512 is trained by the language model 510 as a whole.

[0058] The decoder layers, including the decoding layer 514, are a group of layers included in the language model 510. The decoder layer 514 includes factorized memory blocks 500 and feedforward blocks 516. In at least some embodiments, each decoder layer includes factorized memory blocks. In at least some embodiments, each decoder layer includes only factorized memory blocks. In at least some embodiments, some decoder layers optionally include feedforward blocks, fully connected blocks, etc., or any combination thereof, in addition to factorized memory blocks. In at least some embodiments, some decoder layers include attention blocks or multilayer perceptron (MLP) blocks instead of factorized memory blocks.

[0059] The factorized memory block 500 is a component of the decoding layer 514. In at least some embodiments, the factorized memory block 500 is configured to selectively update a portion of the recursive memory state that forms the basis of the output. In at least some embodiments, the factorized memory block 500 includes memory update and memory merge functions. In at least some embodiments, the factorized memory block 500 is configured to calculate a topic affinity score, update topic vectors based on topic update weight values, and merge the updated topic vectors to generate output token embeddings. In at least some embodiments, the factorized memory block 500 is configured as described in Figure 1.

[0060] The feedforward block 516 is a component of the decoding layer 514. In at least some embodiments, the feedforward block 516 is an optional block within the decoder layer 514. In at least some embodiments, the feedforward block 516 is configured to perform additional processing on the output token embedding. In at least some embodiments, the feedforward block 516 is configured to refine the output projection into an output token embedding.

[0061] The language model head layer 518 is a group of layers included in the language model 510. In at least some embodiments, the language model head layer 518 is configured to decode embedded token vectors into tokens. In at least some embodiments, the language model head layer 518 is configured to construct tokens into natural language responses 519. In at least some embodiments, the language model head layer 518 is configured to decode output token embeddings into natural language responses 519. In at least some embodiments, the language model head layer 518 is generally compatible with the language model. In at least some embodiments, the language model head layer 518 is trained together with the entire language model 510.

[0062] Figure 6 shows an operation flow for assembling and training a language model using factorized memory, according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method for assembling and training a language model using factorized memory. In at least some embodiments, this method is executed by a processor of the device, such as the processor 762 of the device 760 shown in Figure 7, which will be described later.

[0063] In S650, the processor constructs a decoding layer using factorized memory. In at least some embodiments, the processor constructs a decoding layer in which at least some factorized memory blocks are included. In at least some embodiments, the processor constructs a decoding layer in terms of quantity, configuration, and pattern according to user input. In at least some embodiments, the processor includes one or more optional blocks in the decoding layer, such as feedforward blocks and fully connected blocks. In at least some embodiments, the processor constructs a decoding layer in which at least some of the factorized memory blocks include attention blocks or MLPs instead.

[0064] In S652, the processor constructs a token embedding layer, a decoding layer, and a language model head layer. In at least some embodiments, the processor constructs a language model by combining the decoding layer with the input token embedding layer and the output language model head layer. In at least some embodiments, the processor configures the output dimension of the token embedding layer to match the input dimension of the decoding layer. In at least some embodiments, the processor configures the input dimension of the language model head layer to match the output dimension of the decoding layer.

[0065] In S654, the processor selects a value for a configurable parameter. In at least some embodiments, the processor selects the total number of topic vectors, such as m in Equation 1, and the value of Equation 2.

number

[0066] In S656, the processor trains a language model. In at least some embodiments, the processor uses a training set of training samples, calculates a loss according to a loss function, and updates the trainable parameters of the language model according to the calculated loss. In at least some embodiments, the processor trains parameters including topic affinity weights, topic update rate weights, topic merge rate weights, input projection weights, output projection weights, token embedding layer parameters, language model head layer parameters, and any other trainable parameters in the language model. In at least some embodiments, as the language model is trained, the processor divides the hyperspace of token embeddings into multiple topic partitions. In at least some embodiments, as the language model is trained, the processor encodes the centroid of each topic partition as a topic vector, and the topic affinity weights are based on the topic vector.

[0067] In S658, the processor determines whether the accuracy and computational efficiency are acceptable. In response to the processor's determination that the accuracy and computational efficiency are unacceptable, the operation flow returns to S654 to select different values ​​for the configurable parameters. In response to the processor's determination that the accuracy and computational efficiency are acceptable, the operation flow terminates.

[0068] Figure 7 shows an embodiment of a device 760 for language modeling using factorized memory, according to at least some embodiments of the present disclosure. As shown in Figure 7, the device 760 includes a processor 762, memory 763, storage 764, input component 765, output component 766, communication interface 767, and bus 768. The processor 762, as used herein, means any type of computing circuit which may include hardware and software elements. The processor 762 may be embodied as a multicore processor, a single-core processor, a combination of one or more multicore processors and / or one or more single-core processors, or a distributed processing system, etc. The processor 762 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), or another type of processing component.

[0069] Memory 763 includes a non-temporary computer-readable medium. Memory 763 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) for storing information and / or instructions for use by processor 762. Memory 763 includes machine-readable instructions that can be executed by processor 762. When these machine-readable instructions are executed by processor 762, they cause processor 762 to perform one or more method steps of the embodiments described above.

[0070] Storage 764 stores information and / or software related to the operation and use of device 760. For example, storage 764, together with a corresponding drive, may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital multipurpose discs (DVDs), floppy disks, cartridges, magnetic tapes, and / or other types of non-temporary computer-readable media.

[0071] The input component 765 is configured to receive information such as user input. For example, the input component 765 may include, but is not limited to, a touchscreen display, keyboard, keypad, mouse, buttons, switches, and / or a microphone. Additionally or alternatively, the input component 765 may include sensors for sensing information (e.g., a Global Positioning System (GPS), accelerometer, gyroscope, and / or actuator).

[0072] The output component 766 is configured to provide output information from the device 760. For example, the output component 766 may be, but is not limited to, a display, a speaker, an indicator device for an external device, and / or one or more light-emitting diodes (LEDs).

[0073] The communication interface 767 is an interface for establishing communication connections with other devices, such as external or internal devices. Connections via the communication interface 767 may be wired, wireless, or a combination of wired and wireless connections, and may be direct or indirect connections via a communication network existing between device 760 and other devices. In other words, the specifications for the communication interface 767 are not limited.

[0074] Bus 768 functions as an interconnection between the processor 762, memory 763, storage 764, input component 765, output component 766, and the communication interface 767 of the device 760. Bus 768 may include wired or wireless interconnections.

[0075] The number and arrangement of components shown in Figure 7 are provided as an example. In practice, the apparatus 760 may include additional components, fewer components, different components, or components in a different arrangement than that shown in Figure 7. Additionally or alternatively, a set of components of the apparatus 760 (e.g., one or more components) may perform one or more functions that are described as being performed by another set of components of the apparatus 760. Furthermore, one or more method steps described in any of the embodiments may be performed using multiple apparatuses 760 communicating with one another.

[0076] In at least some embodiments, language modeling using factorized memory is performed by a method of operation that includes: calculating a topic affinity score for each topic vector in a plurality of topic vectors based on input token embeddings and a topic affinity weight matrix; updating each topic vector in at least some of the plurality of topic vectors based on the corresponding topic affinity score; and merging at least some of the updated topic vectors in the plurality of topic vectors to generate output token embeddings.

[0077] In at least some embodiments, the update operation of the method includes calculating a topic update rate value for each topic vector in at least some of a plurality of topic vectors, based on topic update rate weights and input token embeddings. In at least some embodiments, updating further includes calculating a topic update weight value for each topic in at least some of a plurality of topic vectors, based on topic update rate values ​​and topic affinity scores. In at least some embodiments, updating further includes calculating an input projection based on an input projection weight matrix and input token embeddings. In at least some embodiments, updating further includes calculating an updated topic vector for each topic vector in at least some of a plurality of topic vectors, based on topic update weight values, input projections, and previous topic vectors. In at least some embodiments, updating further includes storing each updated topic vector in physical memory. In at least some embodiments, updating further includes retrieving a previous topic vector from physical memory corresponding to each topic vector in at least some of a plurality of topic vectors. In at least some embodiments, merging includes calculating a topic merge rate value for each topic vector in at least some of a plurality of topic vectors, based on topic merge rate weights and input token embeddings. In at least some embodiments, merging further includes calculating topic merge weight values ​​for each topic in at least some of the multiple topic vectors based on topic merge rate values ​​and topic affinity scores. In at least some embodiments, merging further includes calculating an output projection based on an output projection weight matrix, updated topic vectors, and topic merge weight values ​​for each topic in at least some of the multiple topic vectors.In at least some embodiments, the method further includes encoding a natural language input into input token embeddings and decoding the output token embeddings into a natural language output. In at least some embodiments, the calculation of each topic affinity score is further based on a topic affinity temperature value. In at least some embodiments, the calculation includes selecting a predetermined number of topic vectors from a plurality of topic vectors having the highest topic affinity score, where the predetermined number of topic vectors is at least some of the plurality of topic vectors.

[0078] In at least some embodiments, language modeling using factorized memory is performed by a device configured to perform operations including: calculating a topic affinity score for each topic vector in a plurality of topic vectors based on input token embeddings and a topic affinity weight matrix; updating each topic vector in at least some of the plurality of topic vectors based on the corresponding topic affinity score; and merging at least some of the updated topic vectors in the plurality of topic vectors to generate output token embeddings.

[0079] In at least some embodiments, the update operation performed by the device includes calculating a topic update rate value for each topic vector in at least some of a plurality of topic vectors, based on topic update rate weights and input token embeddings. In at least some embodiments, updating further includes calculating a topic update weight value for each topic in at least some of a plurality of topic vectors, based on topic update rate values ​​and topic affinity scores. In at least some embodiments, updating further includes calculating an input projection based on an input projection weight matrix and input token embeddings. In at least some embodiments, updating further includes calculating an updated topic vector for each topic vector in at least some of a plurality of topic vectors, based on topic update weight values, input projections, and previous topic vectors. In at least some embodiments, updating further includes storing each updated topic vector in physical memory. In at least some embodiments, updating further includes retrieving a previous topic vector from physical memory corresponding to each topic vector in at least some of a plurality of topic vectors. In at least some embodiments, merging includes calculating a topic merge rate value for each topic vector in at least some of a plurality of topic vectors, based on topic merge rate weights and input token embeddings. In at least some embodiments, merging further includes calculating topic merge weight values ​​for each topic in at least some of the multiple topic vectors based on topic merge rate values ​​and topic affinity scores. In at least some embodiments, merging further includes calculating an output projection based on an output projection weight matrix, updated topic vectors, and topic merge weight values ​​for each topic in at least some of the multiple topic vectors.In at least some embodiments, the operations performed by the device further include encoding a natural language input into input token embeddings and decoding the output token embeddings into a natural language output. In at least some embodiments, the calculation of each topic affinity score is further based on a topic affinity temperature value. In at least some embodiments, the calculation includes selecting a predetermined number of topic vectors from a plurality of topic vectors having the highest topic affinity score, where the predetermined number of topic vectors is at least some of the plurality of topic vectors.

[0080] In at least some embodiments, language modeling using factorized memory is performed by a non-temporary computer-readable medium that includes instructions causing one or more processors to perform operations including: calculating a topic affinity score for each topic vector in a plurality of topic vectors based on an input token embedding and a topic affinity weight matrix; updating each topic vector in at least some of the plurality of topic vectors based on the corresponding topic affinity score; and merging at least some of the updated topic vectors to generate an output token embedding.

[0081] In at least some embodiments, the update operation includes calculating a topic update rate value for each topic vector in at least some of a plurality of topic vectors, based on topic update rate weights and input token embeddings. In at least some embodiments, updating further includes calculating a topic update weight value for each topic in at least some of a plurality of topic vectors, based on topic update rate values ​​and topic affinity scores. In at least some embodiments, updating further includes calculating an input projection based on an input projection weight matrix and input token embeddings. In at least some embodiments, updating further includes calculating an updated topic vector for each topic vector in at least some of a plurality of topic vectors, based on topic update weight values, input projections, and previous topic vectors. In at least some embodiments, updating further includes storing each updated topic vector in physical memory. In at least some embodiments, updating further includes retrieving a previous topic vector from physical memory corresponding to each topic vector in at least some of a plurality of topic vectors. In at least some embodiments, merging includes calculating a topic merge rate value for each topic vector in at least some of a plurality of topic vectors, based on topic merge rate weights and input token embeddings. In at least some embodiments, merging further includes calculating topic merge weight values ​​for each topic in at least some of the multiple topic vectors based on topic merge rate values ​​and topic affinity scores. In at least some embodiments, merging further includes calculating an output projection based on an output projection weight matrix, updated topic vectors, and topic merge weight values ​​for each topic in at least some of the multiple topic vectors.In at least some embodiments, the operation further includes encoding a natural language input into input token embeddings and decoding the output token embeddings into a natural language output. In at least some embodiments, the calculation of each topic affinity score is further based on a topic affinity temperature value. In at least some embodiments, the calculation includes selecting a predetermined number of topic vectors from a plurality of topic vectors having the highest topic affinity score, where the predetermined number of topic vectors is at least some of the plurality of topic vectors. In at least some embodiments, the operation further includes training a language model, which includes a plurality of token embedding layers, at least one decoder layer including a factorized memory block, and a plurality of language model head layers, where the factorized memory block includes trainable parameters including a topic affinity weight matrix, a topic update rate, a topic merge rate, an input projection weight matrix, and an output projection weight matrix. In at least some embodiments, the operation further includes selecting a value for each configurable parameter from at least some configurable parameters, including the total number of topic vectors, the number of updated topic vectors per input embedding, and the topic affinity temperature.

Claims

1. It is a method, The steps include: calculating a topic affinity score for each topic vector among multiple topic vectors based on input token embeddings and a topic affinity weight matrix; A step of updating each topic vector among at least some of the plurality of topic vectors based on the corresponding topic affinity score, A method comprising the step of merging at least some of the updated topic vectors in order to generate output token embeddings.

2. The updating step includes calculating a topic update rate value for each topic vector in at least some of the plurality of topic vectors, based on the topic update rate weight and the input token embedding. The method according to claim 1.

3. The updating step further includes calculating a topic update weight value for each topic among at least some of the plurality of topic vectors, based on the topic update rate value and the topic affinity score. The method according to claim 2.

4. The updating step further includes the step of calculating the input projection based on the input projection weight matrix and the input token embeddings. The method according to claim 3.

5. The updating step further includes the step of calculating an updated topic vector for each topic vector among at least some of the multiple topic vectors, based on the topic update weight value, the input projection, and the previous topic vector. The method according to claim 4.

6. The updating step further includes storing each updated topic vector in physical memory. The method according to claim 5.

7. The merging step includes calculating a topic merge rate value for each topic vector in at least some of the plurality of topic vectors, based on the topic merge rate weight and the input token embedding. The method according to claim 6.

8. The merging step further includes calculating a topic merge weight value for each topic in at least some of the plurality of topic vectors, based on the topic merge rate value and the topic affinity score. The method according to claim 7.

9. The merging step further includes the step of calculating the output projection based on the output projection weight matrix, the updated topic vectors, and the topic merge weight values ​​for each topic among at least some of the plurality of topic vectors. The method according to claim 8.

10. A language model is trained, the language model comprising a plurality of token embedding layers, at least one decoder layer including a factorized memory block, and a plurality of language model head layers, the factorized memory block comprising trainable parameters including the topic affinity weight matrix, the topic update rate, the topic merge rate, the input projection weight matrix, and the output projection weight matrix. The method according to claim 9.

11. Select a value for each configurable parameter among at least several configurable parameters, including the total number of topic vectors, the number of updated topic vectors per input embedding, and the topic affinity temperature. The method according to claim 10.

12. The updating step further includes the step of searching the physical memory for the previous topic vector corresponding to each topic vector among at least some of the plurality of topic vectors. The method according to claim 5.

13. The further step includes encoding natural language input and converting it into an input token embedding. The method according to claim 1.

14. The process further includes the step of decoding the output token embedding and converting it into natural language output. The method according to claim 13.

15. The step of calculating each topic affinity score is further based on the topic affinity temperature value, The method according to claim 1.

16. The calculation step includes selecting a predetermined number of topic vectors from the plurality of topic vectors having the highest topic affinity score, wherein the predetermined number of topic vectors is at least some of the plurality of topic vectors. The method according to claim 1.

17. It is a device, This involves calculating a topic affinity score for each topic vector among multiple topic vectors based on input token embeddings and a topic affinity weight matrix, Updating each topic vector among at least some of the multiple topic vectors based on the corresponding topic affinity score, A device configured to perform an operation including merging at least some of the updated topic vectors to generate an output token embedding.

18. The updating includes calculating a topic update rate value for each topic vector among at least some of the multiple topic vectors, based on the topic update rate weight and the input token embedding. The apparatus according to claim 17.

19. A non-temporary computer-readable medium that responds to execution by one or more processors, This involves calculating a topic affinity score for each topic vector among multiple topic vectors based on input token embeddings and a topic affinity weight matrix, Updating each topic vector among at least some of the multiple topic vectors based on the corresponding topic affinity score, A non-temporary computer-readable medium containing instructions that cause the execution of an action including merging at least some of the updated topic vectors to generate output token embeddings.

20. The updating includes calculating a topic update rate value for each topic vector among at least some of the multiple topic vectors based on the topic update rate weight and the input token embedding. The computer-readable medium according to claim 19.

Images