Efficient embedded table storage and lookup

Abstract

The present disclosure provides systems, methods, and computer program products for providing efficient embedding table storage and lookup in machine learning models. A computer- implemented method can include obtaining an embedding table comprising a plurality of embeddings respectively associated with corresponding indices of the embedding table, individually compressing each particular embedding of the embedding table to allow each respective embedding of the embedding table to be decompressed independently of any other embedding in the embedding table, packaging the embedding table comprising the individually compressed embeddings with a machine learning model, receiving an input for locating an embedding in the embedding table, determining a lookup value to search the indices of the embedding table based on the input, locating the embedding based on searching the indices of the embedding table for the determined lookup value, and decompressing the located embedding independently of any other embedding in the embedding table.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Patent Application No. 17 / 147,844, filed January 13, 2021. The entire contents of U.S. Patent Application No. 17 / 147,844 are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to machine learning. More specifically, this disclosure relates to providing efficient embedded table storage and lookup in machine learning models. Background Technology

[0004] Embedding generally refers to the way objects are projected from a high-dimensional space to a low-dimensional space. For example, image, video, text, and voice data can be represented in an object space such as a two-dimensional or three-dimensional coordinate system. Each data point in the object space can represent an object, and the proximity between each object indicates similarity or relevance. In one example, synonyms can typically be placed more closely together in the object space compared to unrelated or dissimilar words.

[0005] Machine learning models can utilize embedding tables to store and reference embeddings. The embedding table becomes part of the machine learning model, and its size can be quite large. For example, an embedding table of dictionary words from one or more languages ​​could have hundreds of thousands or even millions of entries. In particular, embedding tables can significantly increase the amount of memory and processing power required to run machine learning models. Therefore, serving large models on a large scale becomes increasingly difficult. Furthermore, many types of computing devices, including mobile devices, often lack sufficient computing resources to run large models. Summary of the Invention

[0006] Various aspects and advantages of the embodiments of the present invention will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0007] One example aspect of this disclosure relates to a system for providing efficient embedding table storage and lookup for a machine learning model, for example, by: receiving an embedding table comprising multiple embeddings correspondingly associated with corresponding indexes of the embedding table; individually compressing each particular embedding of the embedding table to allow each corresponding embedding of the embedding table to be decompressed independently of any other embedding in the embedding table; packaging the embedding table comprising the individually compressed embeddings together with the machine learning model; receiving input for locating embeddings in the embedding table; determining a lookup value based on the input to search the indexes of the embedding table; locating the embeddings based on the search of the indexes of the embedding table for the determined lookup value; decompressing the located embeddings independently of any other embeddings in the embedding table; and processing the decompressed embeddings in conjunction with running the machine learning model.

[0008] Other aspects of this disclosure relate to various apparatuses, non-transitory computer-readable media, computer-implemented methods, user interfaces, and electronic devices. These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0009] The specification provides a detailed discussion of embodiments for those skilled in the art, with reference to the accompanying drawings, in which:

[0010] Figure 1A A block diagram of an example system for providing efficient embedded table storage and lookup in a machine learning model, according to an example embodiment of the present disclosure, is depicted.

[0011] Figure 1B A block diagram depicts an example embedded table for providing efficient embedded table storage and lookup in a machine learning model, according to an example embodiment of the present disclosure.

[0012] Figure 2 A flowchart is depicted illustrating an example method for providing an efficient embedded table storage for a machine learning model, according to an example embodiment of the present disclosure.

[0013] Figure 3 A block diagram illustrating an example of providing an efficient embedded table storage for a machine learning model, according to an example embodiment of the present disclosure.

[0014] Figure 4 A flowchart is depicted illustrating an example method for providing efficient embedded table storage and lookup for machine learning models according to an example embodiment of the present disclosure.

[0015] Figure 5 A block diagram illustrating an example of providing efficient embedded table storage and lookup for a machine learning model according to an example embodiment of the present disclosure.

[0016] Figure 6A A block diagram of an example computing system that performs efficient embedded table storage and lookup according to an example embodiment of the present disclosure is depicted.

[0017] Figure 6B A block diagram of an example computing device for performing efficient embedded table storage and lookup according to an example embodiment of the present disclosure is depicted.

[0018] Figure 6C A block diagram of an example computing device for performing efficient embedded table storage and lookup according to an example embodiment of the present disclosure is depicted.

[0019] The repeated reference numerals across multiple figures are intended to identify the same features in various embodiments. Detailed Implementation

[0020] Overview

[0021] Generally, this disclosure relates to providing efficient embedding table storage and retrieval for machine learning models. The examples described in this disclosure are capable of efficiently storing and retrieving embedding tables used with machine learning models. Such exemplary implementations offer advantages and improvements over existing methods, including but not limited to improved scalability, enhanced performance, and reduced computational requirements compared to existing methods.

[0022] Embedding tables are used as components of machine learning models. For example, embedding tables can be used in machine learning models related to Natural Language Processing (NLP) or various other implementations. Embedding tables help construct abstract relationships between objects (e.g., words, phrases, images, songs, movies, etc.) and include embeddings that project such objects from a high-dimensional space to a lower-dimensional space. For example, words, phrases, images, songs, movies, or any other type of object can be represented in an object space, such as a two- or three-dimensional coordinate system, where proximity between objects represents relevance between them. In various examples, the object space can be of any number of dimensions and is not limited to two or three dimensions. For example, an object space can have more than three dimensions, more than ten dimensions, more than one hundred dimensions, or generally any number of dimensions. Furthermore, embedding tables can be trained within a machine learning model and can be reused across different machine learning models.

[0023] Embedding tables are stored as part of a machine learning model and can become very large. For example, a machine learning model may utilize one or more embedding tables, which can include thousands, millions, or any number of entries. In fact, an embedding table can represent a significant or overwhelming portion of the size of a machine learning model. Furthermore, traditional machine learning platforms and operations require computing devices to load the full-size embedding table of the machine learning model into memory at once. Serving large machine learning models on a large scale becomes increasingly complex and expensive due to the significant memory and computing resources required. Moreover, these memory and processing requirements often exceed the limited capabilities of many types of mobile computing devices, which typically lack the computer hardware needed to process large models.

[0024] To address and resolve these issues, this disclosure provides examples of efficient embedding table storage and lookup for machine learning models. In some examples of this disclosure, a computational system: performs operations to generate or otherwise obtain an embedding table comprising multiple embeddings correspondingly associated with a corresponding index of the embedding table; individually compresses each specific embedding of the embedding table to allow each corresponding embedding of the embedding table to be decompressed independently of any other embedding in the embedding table; packages the embedding table, including the individually compressed embeddings, together with a machine learning model; obtains input for locating embeddings in the embedding table; determines a lookup value based on the input of the search index of the embedding table; searches the index of the embedding table based on the determined lookup value to locate the embedding; decompresses the located embedding independently of any other embedding in the embedding table; and processes the decompressed embedding as part of running a machine learning model associated with the embedding table.

[0025] The systems, methods, and computer program products described herein offer numerous technical effects and benefits. For example, examples of this disclosure describe specialized embedded table data structures and associative operations that allow a computing system to access specific entries of an embedded table while running a machine learning model without loading the entire embedded table into memory. In this way, the examples described in this disclosure provide efficient ways to store and retrieve data in embedded tables and associative machine learning models, utilizing fewer computational resources (e.g., less processing power, less memory usage, less power consumption, less storage space, etc.) compared to conventional operations.

[0026] The systems, methods, and computer program products described in the examples of this disclosure are particularly well-suited for use in embedding tables in machine learning models associated with Natural Language Processing (NLP). Nevertheless, the methodologies described herein can be applied to a wide range of technical applications using embedding tables, including but not limited to: image recognition, image classification, image captioning, scene segmentation, object detection, action recognition, semantic segmentation, speech detection, speech translation, speaker identification, language prediction, text classification, face recognition, etc. Input data can include, for example, one or more of any type of image, video, text, speech, audio, sensor data, and / or any other type of data. Furthermore, the examples of this disclosure are not limited to use with embedding tables in machine learning models and can generally be used with embedding tables or any other similar data structures.

[0027] Exemplary embodiments of this disclosure will now be discussed in more detail with reference to the accompanying drawings.

[0028] Provides an example system for efficient embedded table storage and lookup.

[0029] Figure 1AA block diagram of an example system for providing efficient embedded table storage and lookup in a machine learning model, according to an example embodiment of the present disclosure, is depicted.

[0030] System 100 includes a computing device 110, a model 120 utilizing one or more embedding tables 130, a compression operation 140, and a compressed model 150 including one or more compressed embedding tables 160. System 100 also includes at least one network 170, a computing device 112 including a compressed model 152 and a compressed embedding table 162, and a computing device 114 including a compressed model 154 and a compressed embedding table 164.

[0031] Computing device 110 can typically be any type of computer device, such as a personal computer (PC), laptop computer, mobile phone, tablet computer, server computer, wearable computing device, or any other type of computing device. Computing device 110 can run an operating system (OS) that manages the hardware and software of the device. Computing device 110 can also run one or more machine learning platforms (not shown) that provide computer software, software development tools, software libraries, and / or application programming interfaces (APIs) for developing, training, testing, executing, and / or deploying computer software applications and associated data structures (e.g., machine learning models), configured to perform operations related to machine learning and artificial intelligence. Examples of machine learning platforms may include, but are not limited to, those provided. and Machine learning platform. Computing device 110 may typically include one or more machine learning models, such as model 120.

[0032] Model 120 typically refers to a specialized data structure configured and trained to perform one or more types of machine learning tasks, such as prediction, detection, classification, identification, etc. In the example, Model 120 can be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some example machine learning models can fully utilize attention mechanisms, such as self-attention. For example, some example machine learning models can include multi-head self-attention models (e.g., transformer models). Further machine learning models (e.g., Model 120) can include and utilize one or more embedding tables 130 for assisting in performing one or more types of machine learning tasks.

[0033] An embedding table (130) typically refers to a table or any other type of data structure used to index and store a collection of embeddings associated with a group of objects. An index is typically a unique value or key that can be used to find an embedding associated with a particular input value. In some examples, each embedding may typically represent or otherwise include a collection of one or more numeric values, such as a numeric vector, which projects a particular object onto an object space (e.g., a two-dimensional, three-dimensional, or any other type of object space with arbitrary dimensions).

[0034] Embedding table 130 can typically be used to construct and store complex relationships between various types of inputs, such as words, phrases, images, videos, text, and speech data. In one example, embedding table 130 can be used to store embeddings. For example, each word in the set of words can be associated with a learned representation defined as a numerical value in a vector determined based on the word's meaning. In another example, each word in the set of words can be associated with a learned representation based on the word's origin or the language of each corresponding word, stored as a set of numerical values ​​for each word. For example, word embeddings can often be associated with natural language processing, where words, phrases, and / or any other features of word content are mapped to real-valued vectors utilized by machine learning models.

[0035] Typically, embeddings can be determined and used for any type of input data. As such, embeddings are not limited to words and text. Furthermore, the embedding table 130 and associated embeddings can be trained, for example, separately from and / or as part of a machine learning model. Moreover, the embedding table can be reused across different machine learning models.

[0036] The size of the embedding table 130, based on a large input set, can become extremely large. For example, an embedding table 130 based on words in a vocabulary, such as a dictionary from a single language or a collection of dictionaries in different languages ​​(e.g., 2, 10, 50, 100, or more languages), may account for a significant or even almost all of the size of the machine learning model. For example, some embedding tables 130 may include millions or even billions of entries. In this way, the size of model 120 can grow to hundreds of megabytes or even gigabytes. Furthermore, such a large model 120 becomes increasingly difficult to scale and is generally not feasible to run on computing devices with limited computing resources. For example, many types of mobile devices, including wearable devices, edge devices, medical devices, and / or other types of portable computing devices, typically lack or otherwise require the computer memory and computer hardware resources needed to run model 120 utilizing a large embedding table 130.

[0037] Typically, model 120 and embedding table 130 can be compressed. For example, machine learning platforms often provide some form of platform-supported compression that supports compressing the entire model 120 and each table in the associated embedding table 130 as a whole. However, to run such a model, the computer system must include and contribute sufficient memory and processing power to run the entire model 120 and the associated full-size decompressed embedding table 130. In this way, standard compression does not address reducing computational requirements to help scale model 120 with a large embedding table 130 or to run model 120 with a large embedding table 130 on devices with limited, minimal, or unavailable hardware resources.

[0038] In one example, one or more compression operations 140 may be performed to provide efficient storage and lookup of the embedding table 130 for a machine learning model. For example, one or more compression operations 140 may be used to compress individual rows in the embedding table 130 to allow a computing device to run model 120 without loading and processing the entire decompressed embedding table 130 in memory. In some examples, model 120 may also be compressed. For example, aspects of model 120 may be pruned, optimized, and / or compressed to produce a compressed model 150. In some examples, model 120 and / or compressed model 150 may typically be represented, stored, analyzed, updated, and / or run as a graph or graph data structure.

[0039] Compression operation 140 can be performed independently of any underlying machine learning platform and / or using a compression scheme unavailable from the underlying machine learning platform. For example, compression operation 140 is typically not supported by or otherwise provided by the underlying machine learning platform. Furthermore, the association lookup and decompression operations corresponding to compression operation 140 can be included within and performed by model 120 itself, without using or referencing the underlying machine learning platform. In this way, association lookup and decompression operations for accessing individual compressed rows of the association embedding table 130 can be self-contained and used as part of running model 120, independent of any underlying machine learning platform. Therefore, model 120 with a large embedding table 130 can be run with significantly reduced memory and computational resources, thereby improving overall scalability and allowing model 120 with a large embedding table 130 to run on devices with limited hardware capabilities.

[0040] In one example, one or more compression operations 140 are used to compress model 120 and / or associated embedding table 130. For example, the computing system may receive, generate, or otherwise obtain one or more embedding tables 130 associated with machine learning model 120. The computing system then uses compression operation 140 to individually compress each specific embedding of the one or more embedding tables 130 to allow each corresponding embedding of a specific embedding table 130 to be decompressed independently of any other embedding in that specific embedding table 130. The computing system can then use compression operation 140 to package one or more compressed embedding tables 160, each with individually compressed embeddings, together with the associated machine learning model. In some examples, compressed embedding table 160 is packaged together with compressed model 150. For example, one or more aspects of model 120 may be pruned, optimized, and / or compressed using compression operation 140 to generate compressed model 150. In some examples, compressed embedding table 160 may also be packaged together with model 120, which itself has not yet been compressed.

[0041] In one example, compression model 150 and associated compression embedding table 160 are packaged together and provided to one or more computer systems for execution. For example, compression model 150 and compression embedding table 160 may be stored, deployed, transmitted, or otherwise moved to one or more other locations that allow one or more computer systems to run compression model 150 with compression embedding table 160 and accordingly utilize associated operations. In some examples, compression model 150 and compression embedding table 160 may be locally stored for execution by the same computing device (e.g., computing device 110) running compression operation 140. In some examples, compression model 150 and compression embedding table 160 are transmitted via network 170 to one or more other computing devices 112, 114.

[0042] Network 170 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), or any combination thereof. In the example, network 170 may include the Internet, one or more intranets, wired networks, wireless networks, and / or other suitable types of communication networks. Network 170 may also include wireless telecommunication networks (e.g., cellular networks), such as the Internet, adapted to communicate with other communication networks. Furthermore, network 170 may include one or more short-range wireless networks.

[0043] Computing device 112 and computing device 114 can each be any type of computer device, such as a personal computer (PC), laptop computer, mobile phone, tablet computer, server computer, wearable computing device, or any other type of computing device. Computing device 112 and computing device 114 can also each run an operating system (OS) and can each run one or more machine learning platforms of the same or different types.

[0044] Computing device 112 includes a compressed model 152 with an associated compressed embedding table 162. Computing device 114 includes a compressed model 154 with an associated compressed embedding table 164. Computing model 152 and compressed model 154 may be the same or different. Computed embedding table 162 and compressed embedding table 164 may also be the same or different, regardless of whether compressed model 152 and compressed model 154 are the same.

[0045] In the example, each computing device 110, 112, 114 can run a corresponding compressed model 150, 152, 154, and each compressed model 150, 152, 154 uses a corresponding compressed embedding table 160, 162, 164. In the example, a computing device such as computing device 112 receives, loads, and / or otherwise obtains compressed embedding table 162. For example, compressed embedding table 162 may be associated with a compressed model 160 stored on computing device 112 or otherwise made available to computing device 112. In one example, compressed model 160 itself can be decompressed into model 120, which utilizes compressed embedding table 162. In some examples, model 120 or compressed model 160 may be run by one or more applications executed by computing device 112.

[0046] In one example, model 120 or compression models 150, 152, 154 may provide one or more lookup and decompression operations (not shown) configured to perform lookups and decompressions on corresponding compressed embedding tables 160, 162, 164, including individually compressed embeddings. For example, computing device 112 may run compression model 152, which receives input for locating embeddings in the corresponding compressed embedding table 162. Compression model 152 may then determine lookup values ​​for searching indices in the corresponding compressed embedding table 162 to locate the embedding associated with the input. For example, compression model 152 may perform one or more operations on the input or some derivation thereof based on a lookup value determined by an index in the compressed embedding table 162 for input-based location mapping.

[0047] In one example, compression model 152 uses the determined lookup value to search the index of compressed embedding table 162 to locate the individually compressed embedding. Compression model 152 then decompresses the individually compressed embedding independently of any other embeddings in the embedding table to obtain the decompression value associated with the located individually compressed embedding. In this way, compression model 152 can perform the lookup and decompression of individually compressed embeddings within the corresponding compressed embedding table 162 based on specific input, without decompressing and / or loading any other irrelevant embeddings or parts of compressed embedding table 162 into memory.

[0048] In the various examples, compressed embedding table 162 stores individually compressed embeddings and indexes, where each individually compressed embedding corresponds to one of the indexes. Furthermore, individually compressed embeddings can typically include or refer to any associated compressed storage unit, including but not limited to individually compressed records, entries, rows, tuples, and / or any other type of logical or physical storage unit in compressed embedding table 162 that holds individually compressed embeddings.

[0049] Figure 1B A block diagram of an example embedding table for providing efficient embedding table storage and lookup in a machine learning model, according to an example embodiment of the present disclosure, is depicted. The example embedding table 180 includes an embedding table data structure 182, an embedding table index 184, an embedding 186, input values ​​188, and a process 190 for separately decompressed embeddings.

[0050] In the example, the embedded table data structure 182 may be an embedded table that includes a separately compressed embedded 186. For example, the embedded table data structure 182 may include multiple distinct rows or entries, each row or entry having an embedded table index 184 that references the corresponding separately compressed embedded 186. In one example, each row in the embedded table data structure 182 includes an embedded table index 184 (index_A, index_B, index_C, etc.) that references the corresponding separately compressed embedded 186.

[0051] Typically, the embedding table index 184 can be a unique value or key used to find embeddings 186 that include one or more values. In some examples, each embedding 186 includes a collection of one or more numeric values, such as a numeric vector, which projects a particular object into an object space (e.g., a two-dimensional, three-dimensional, or any other type of object space with arbitrary dimensions). For example, each embedding 186 in the embedding table data structure 182 can include a group of numeric values, where each numeric value corresponds to a weight associated with a particular dimension (e.g., a measurable attribute). Typically, embedding values ​​can be set by default, provided from another source, generated as a result of training, updated as a result of training and / or retraining, etc.

[0052] In one example, each embedding 186 in the embedding table data structure 182 includes a numerical weight associated with the "Animal" dimension of the input value 188, a numerical weight associated with the "Feline" dimension of the input value 188, and so on. In this way, in the embedding table data structure 182, the set of values ​​for each of the dimensions ("Animal", "Feline", "Canine", "Plural", and "Color") in a particular row typically represents an embedding 186 associated with a particular input value 188. For example, row #2 of the embedding table data structure 182 represents an embedding 186 including the numerical set (-0.7, -0.5, -0.3, 0.9, 0.7), each value associated with the corresponding dimension of the input value 188 corresponding to "Houses". Furthermore, the embedding 186 of row #2 of the embedded table data structure can be located and referenced in various embedded table indexes 184 using "index_B", for example, based on the mapping between "index_B" and the input value 188 of "house".

[0053] In the example, each embedding 186 in each row of the embedded table data structure 182 is compressed independently of any other embedding in any other row. For example, embedding 186 in row #1 of the embedded table data structure 182 can be compressed separately and separated from all other embeddings 186 in the other rows #2 through #9 of the embedded table data structure 182. Similarly, each of the other embeddings 186 can also be compressed independently of any other embedding 186 in the embedded table data structure 182. In this way, each individually compressed embedding 186 of the embedded table data structure 182 can be accessed separately via the corresponding embedded table index 184 and decompressed independently of the other embeddings 186, for example, to avoid decompressing, loading, and / or processing the entire embedded table data structure 182 in memory.

[0054] In the example, each embedded table index 184 of the embedded table data structure 182 is generated based on the input value 188. For example, the "index_A" of the embedded table data structure 182 can be generated based on the input value 188 of "Birds". In various examples, each input value 188 is deterministically mapped to a corresponding embedded table index 184. For example, one or more hashes and / or other operations are applied to the input value 188 to generate the corresponding embedded table index 184. In this way, each embedded table index 184 can be determined based on a specific input value 188 and used to locate and individually decompress the associated embedded 186 in the embedded table data structure 182.

[0055] In the example, the embedding table index 184 is used to find and individually decompress the associated embedding 186 within the embedding table data structure 182. In one example, one or more input values ​​188 are received for processing. For example, input values ​​188 such as "house," "cat," "kitten," and "dog" can be received. Each of these input values ​​188 can be processed individually, for example, using hashing and / or one or more other operations to determine the corresponding embedding table index 184. For example, the "house" input value 188 can be processed to determine the embedding table index 184 of "index_B" for use when searching the embedding table index 184 of the embedding table data structure 182 to find the embedding 186 associated with "house."

[0056] In the example, the embedding table index 184 for “index_B” is found at row #2 of the embedding table data structure 182, where the associated embedding 186 is accessed and decompressed separately from other embeddings in the embedding table data structure 182. The input values ​​188 for “cat,” “kitten,” and “dog” can be processed similarly to locate and decompress specific embeddings 186 in the embedding table data structure 182 individually. In various examples, the processing 190 of the individually decompressed embeddings may include, but is not limited to, averaging, joining, exploiting, returning, forwarding, and / or performing any other operations involving the individually decompressed embeddings 186 obtained from the embedding table data structure 182.

[0057] Example methods for efficient embedding of table storage and lookup in machine learning models

[0058] Figure 2 A flowchart depicts an example method for providing an efficient embedded table storage for a machine learning model, according to an example embodiment of this disclosure. Although Figure 2 The steps performed in a specific order are described for illustrative and discussion purposes only; however, the method disclosed herein is not limited to the specifically described order or arrangement. Without departing from the scope of this disclosure, the individual steps of method 200 can be omitted, rearranged, combined, and / or adjusted in various ways.

[0059] At 202, the computing system obtains an embedding table associated with the machine learning model. In the example, computing system 110 obtains one or more embedding tables 130. For example, computing system 110 may receive embedding table 130 or the location of embedding table 130 as input. Computing system 110 may also automatically detect one or more embedding tables 130, for example, based on analysis association model 120 or compression model 150 associated with embedding table 130.

[0060] In the example, computing system 110 may receive input, such as objects and lists or sets of associated embeddings for generating one or more embedding tables 130. For example, computing system 110 may receive a list or set of words from a vocabulary, such as dictionary words for a specific language. Computing system 110 can then use the list or set of words to generate the associated embedding table 130, for example, based on existing or known embeddings, default or general embeddings, and / or any type of embedding that is typically associated with each corresponding entry in the embedding table 130. In some examples, computing system 110 receives multiple lists or sets of different words, each list or set of words associated with a different language (e.g., 2, 10, 100, or more languages), where the list or set of words from each specific language is used to generate the corresponding embedding table 130. In such an example, each list or set of words from a specific language will have its own specific embedding table 130.

[0061] In the example, computing system 110 may generate, transform, or otherwise update one or more indexes of an existing embedding table 130. For example, one or more entries in embedding table 130 obtained or received by computing system 110 may already have corresponding index values ​​associated with the corresponding embeddings. In one example, existing index values ​​of embedding table 130 associated with a vocabulary of words may be the actual word corresponding to a particular embedding or some of its derivatives (e.g., untuples, bigrams, triples, etc. of words). In this way, computing system 110 may determine the index value of each embedding based on existing index values ​​present in embedding table 130 or existing index values ​​from one or more external sets or lists of values ​​(e.g., words, objects, or any derivatives thereof) that do not exist in embedding table 130.

[0062] In the example, computing system 110 determines one or more index values ​​of embedding table 130 based on applying a hash operation to each of one or more values ​​that will be associated with a specific embedding from embedding table 130. In this way, computing system can determine new index values ​​based on existing index values ​​from existing embedding table 130 or based on a list or set of objects used to generate and / or expand embedding table 130.

[0063] In the examples, computation system 110 generates, transforms, or otherwise updates index values ​​of embedded table 130 in association with performing compression operation 140. In various examples, one or more operations—including, but not limited to, hash operations—are applied to input values ​​to generate indexes of embedded table 130, each index being deterministically mapped to a corresponding input. Furthermore, any associated hash operations may use a hash seed. In some examples, the one or more operations used to generate the indexes of embedded table 130 may map multiple distinct input values ​​to the same index of embedded table 130, thereby reducing the total number and size of entries in embedded table 130. In some examples, the one or more operations used to generate the indexes of embedded table 130 may typically include one or more operations prior to and / or after a hash operation performed on any particular value.

[0064] At 204, the computing system individually compresses each specific embedding in the embedding table to allow for individual decompression of each specific embedding independently of other embeddings in the embedding table. In the example, computing system 110 uses compression operation 140 to individually compress each of one or more embeddings in embedding table 130. For example, compression operation 140 can be used to individually compress each corresponding embedding in embedding table 130. In various examples, individual compression of each specific embedding typically allows each embedding or embedding entry to be decompressed and accessed individually, e.g., without decompressing and / or loading any other unrelated embeddings from embedding table 130 into memory.

[0065] In the examples, compression operation 140 is performed independently of any underlying machine learning platform. For example, compression operation 140 may be performed by software that is separate from and unrelated to the underlying machine learning platform. In various examples, such software may be provided, for example, as a software application, script, utility, library, and / or tool that provides specialized operations not provided by the machine learning platform. Furthermore, compression operation 140 may typically provide one or more different compression schemes that are not available from or otherwise supported by the underlying machine learning platform. In some examples, compression operation 140 may compress the embeddings of embedding table 130 independently, for example, based on the selection of one or more available compression schemes, which may include, but is not limited to, quantization compression, k-means compression, and / or pruning compression. However, any compression scheme or operation of this type may typically be provided, including custom and / or later-known compression operations, to generate compressed embedding table 160 independently of and separately from the underlying machine learning platform.

[0066] In the example, compression operation 140 can compress each embedding of embedding table 130 individually, such that each embedding entry in the embedding table is processed independently of the other embeddings in embedding table 130. In this way, model 120 can perform a self-contained lookup and decompression of each specific embedding entry in embedding table 130 in a single operation without using an underlying machine learning platform. In some examples, downstream lookup and decompression operations for compressing embedding table 160 are performed automatically by model 120 itself. In some examples, such downstream lookup and decompression operations for compressing embedding table 160 may also be supported, for example, wholly or partially by a lightweight software framework or library associated with compression operation 140.

[0067] In some examples, model 120 and / or compressed model 150 can be modified or adjusted to decompress specific embeddings independently of other embeddings in compressed embedding table 160, for example, when models 120, 150 are packaged together with compressed embedding table 160 having separate compressed embeddings. For example, such models 120, 150 can be modified, adjusted, configured, or reconfigured (e.g., via compression operation 140) to directly decompress specific separately compressed embeddings as part of a corresponding lookup operation performed on compressed embedding table 160.

[0068] In various examples, the underlying machine learning platform may typically fail to recognize or be aware of the association lookup and decompression of specific embeddings in compressed embedding table 160 or performed by model 120 or compressed model 150. In this way, it is possible to perform individual decompression of specific embeddings without activating the underlying machine learning platform's operations; for example, the entire compressed embedding table 160 could be decompressed immediately in other ways, regardless of whether the underlying compression scheme is actually understood or supported by the underlying machine learning platform.

[0069] At 206, the computing system packages an embedding table with individually compressed embeddings along with a machine learning model. In the example, computing system 110 packages a compressed embedding table 160 along with an associated model 120 or a compressed model 150. For example, computing system 110 may package a compressed model 150 and an associated compressed embedding table 160 together, such that the compressed embedding table 160 is combined with or becomes part of the compressed model. In various examples, the compressed model 150 and the compressed embedding table 160 may be stored, deployed, transferred, moved, and / or run on each of one or more other different computing devices. For example, the compressed model 150 and the compressed embedding table 160 may be stored and run on computing device 110. Furthermore, one or more instances of the compressed model 150 and the compressed embedding table 160 may be created or copied and then transferred to run on each of one or more other computing devices 112, 114.

[0070] Figure 3A flowchart depicts an example method for providing an efficient embedded table storage for a machine learning model, according to an example embodiment of this disclosure. Although Figure 3 The steps performed in a specific order are described for illustrative and discussion purposes only; however, the method of this disclosure is not limited to the specifically described order or arrangement. The various steps of method 300 may be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of this disclosure.

[0071] At 302, the computing system receives input for generating a set of values ​​for each of one or more embedding features. In the example, computing system 110 receives or otherwise obtains data for generating one or more embedding tables 130. For example, a corresponding embedding table 130 may be created for each of one or more embedding features extracted from or otherwise associated with text, audio, or visual input data.

[0072] In some examples, embedded features can be determined or extracted directly from the input based on the analysis input. For example, it is possible to deduce a sequence of consecutive letters or words from words in text or speech. In some examples, embedded features can also be provided or otherwise associated with the input. For example, it is possible to provide audio or visual input for one or more categories or classification descriptors. In one example, video input can be classified in one or more ways, including by rating, genre, era, popularity, studio, etc.

[0073] In the examples, one or more embedding features can be extracted or determined based on the input, wherein each set of values ​​extracted or determined for the embedding features is used to generate a corresponding embedding table 130 for use by a machine learning model. For example, a first set of one or more values ​​for a specific embedding feature, such as a unary tuple for a language model, can be extracted or determined from the input data. A second set of one or more values ​​for different embedding features, such as a binary tuple for a language model, can then be extracted or determined from the input data. Furthermore, one or more other sets of values, such as triples, quadruples, quintuples, etc., can be generated for each additional embedding feature. Furthermore, each of the sets of values ​​associated with a corresponding embedding feature (e.g., unary tuple, binary tuple, triple, etc.) can then be used to generate a corresponding embedding table 130 for use with a machine learning model 120. For example, a first embedding table 130 can be generated for a set of values ​​associated with a unary tuple, a second embedding table 130 can be generated for another set of values ​​associated with a binary tuple, and so on. In various examples, such embedding table 130 can be used with one or more types of machine learning models, including but not limited to a model 120 related to natural language processing (NLP).

[0074] At 304, the computation system generates an embedding table 130 for each of one or more embedding features associated with the input. In an example, the computation system 110 may generate the embedding table 130 as a two-dimensional data structure, which includes at least multiple indexes and multiple embeddings, where each embedding is referenced by a specific one in the index. In some examples, the embedding table 130 may store, reference, or utilize other data and / or features. For example, each row of the embedding table may include one or more fields in addition to the embedding table index and the embedding corresponding to that index.

[0075] In various examples, the computing system can generate one or more embedding tables 130 by creating a new embedding table 130 based on input data, data determined or extracted from the input data, or an existing embedding table 130. In some examples, the computing system 110 can generate embedding tables 130 by creating a new embedding table 130 and inserting associated data accordingly, creating a copy of an existing embedding table 130 and updating the copy accordingly, and / or directly performing operations on the existing embedding table 130 to transform, expand, edit, or otherwise update the data in the existing embedding table 130 using new information. In the examples, the computing system 110 generates one or more corresponding embedding tables 130, each corresponding to a specific embedding feature of the input, for example, based on the example operations performed at 306, 308, and 310, as discussed below.

[0076] At 306, the computation system determines the index of each in the embedding table. In the example, computation system 110 extracts and / or determines sets of values ​​for each of several different embedding features, for example, based on the analytical input. Computation system 110 then refers to the corresponding sets of values ​​associated with a particular embedding feature to generate the corresponding embedding table 130.

[0077] In the example, computational system 110 processes words from a vocabulary of a specific language. Computational system 110 analyzes the words and extracts or determines one or more embedding features associated with each word. For example, computational system 110 may utilize n-tuple embedding features to generate an embedding table for use in a natural language processing (NLP) machine learning model 120. In the example, computational system 110 generates sets of values ​​for each of the one or more embedding features based on the word input. For example, a set of letter values ​​can be generated for a one-tuple embedding feature. A second set of letter combinations can be generated for a two-tuple embedding feature. Furthermore, a third set of letter combinations can be generated for a three-tuple embedding feature.

[0078] In the example, computation system 110 determines the index of a first embedding table corresponding to the unary embedding feature by performing one or more hashing and / or other operations on each value in the set of values ​​associated with the unary embedding feature, to generate an embedding table index that maps to the original unary values. Continuing the example, computation system 110 determines the index of a second embedding table corresponding to the binary embedding feature by performing one or more hashing and / or other operations on each value in the set of values ​​associated with the binary embedding feature, to generate an embedding table index that maps to the original binary values. Furthermore, computation system 110 determines the index of a third embedding table corresponding to the triplet embedding feature by performing one or more hashing and / or other operations on each value in the set of values ​​associated with the triplet embedding feature, to generate an embedding table index that maps to the original triplet values.

[0079] In various examples, each defined index in embedding table 130 is mapped to a value associated with the corresponding embedding feature. Furthermore, each index can be used to look up the associated embedding in a specific embedding table 130 and / or compressed embedding table 160. In some examples, multiple distinct values ​​can be mapped to the same index and associated embedding based on mappings provided by one or more hashes and / or other operations, thus reducing the total number and size of entries in embedding table 130.

[0080] At 308, the computation system stores the indexes and corresponding embeddings in each respective embedding table. In the example, computation system 110 stores the indexes determined based on the input and their corresponding embeddings in embedding table 130. For example, computation system 110 may store a specific set of indexes and corresponding embeddings associated with an embedding feature in one embedding table, while storing another specific set of indexes and corresponding embeddings of another embedding feature in a different embedding table 130. Furthermore, one or more such embedding tables 130 may be used individually or in association with any number of machine learning models 120. In various examples, one or more embeddings of the embedding table may be provided, predetermined, defaulted, learned as part of training a specific embedding table 130, learned as part of training a specific embedding table 130 using model 120, and / or otherwise determined and updated.

[0081] At 310, the computation system compresses each specific row of each embedding table to allow decompression of each specific row of the corresponding embedding table independently of any other row in the corresponding embedding table. In the example, computation system 110 compresses each embedding in embedding table 130 to be individually located and decompressed independently of any other embedding. For example, computation system 110 can compress individual embeddings for each of one or more different embedding tables 130. In the example, each of multiple different embedding tables can be stacked and combined into a single unit (e.g., a tensor), which can later be unpacked when association model 120 or compression model 150 performs an operation (e.g., a lookup operation) involving one or more of the corresponding compressed embedding tables 160. In one example, such embedding table stacking can be provided by compression operation 140. Furthermore, destacking of embedding tables can be performed, for example, by model 120 or compression model 150.

[0082] In the example, the embeddings of a specific embedding table 130 can be compressed individually to generate a compressed embedding table 160, regardless of whether such compression operation and / or associated compression scheme are available or supported by the underlying machine learning platform of any source computing machine and / or any target computing machine. In this way, generally any type of compression, including custom and / or later-known compression operations, can be used to compress the embeddings of embedding table 130 individually to generate a compressed embedding table 160 that can be used independently of and separately from any underlying machine learning platform by model 120 or compressed model 150. In some examples, the embeddings of embedding table 130 can be compressed using custom quantization compression, custom pruning compression, custom k-means compression, and / or any custom or specialized compression that is not available to or supported by the machine learning platform.

[0083] At 312, the computing system transmits each embedding table, comprising individual compressed rows of associated models, to one or more computing devices. In the example, computing system 110 uses compression operation 140 to compress model 120 and generate compressed model 150, which is packaged with one or more compressed embedding tables 160. In some examples, generating the compressed model and / or packaging the compressed model with one or more compressed embedding tables 160 may involve one or more operations that configure compressed model 150 to perform individual decompression of embeddings based on a compression scheme used to individually compress the embeddings of compressed embedding tables 160. For example, compressed model 150 may be adapted or otherwise configured or reconfigured to directly decompress specific individually compressed embeddings based on a specific compression scheme not supported by the underlying machine learning platform, as part of a corresponding lookup operation performed on compressed embedding table 160.

[0084] In the example, compression model 150 and associated compression embedding table 160 are packaged together and provided to one or more computer systems for execution. For example, compression model 150 and compression embedding table 160 may be transferred from a computing device 110, deployed, or otherwise transferred to one or more other computing devices 112, 114 for execution accordingly.

[0085] Figure 4 A flowchart depicts an example method for providing efficient embedded table storage and lookup for machine learning models according to an example embodiment of this disclosure. Although Figure 4 The steps performed in a specific order are described for illustrative and discussion purposes only; however, the method of this disclosure is not limited to the specifically described order or arrangement. Without departing from the scope of this disclosure, the individual steps of method 400 can be omitted, rearranged, combined, and / or adjusted in various ways.

[0086] At 402, the computing system obtains an embedding table that includes individually compressed embeddings. In an example, the computing system, such as computing system 112, obtains a compressed embedding table 162 that includes an index and individually compressed embeddings. For example, the index of compressed embedding table 162 may typically reference one of several individually compressed embeddings that can be decompressed independently of any other embeddings in compressed embedding table 162. In some examples, computing system 112 may receive one or more compressed embedding tables 162 packaged together with compression model 152. Similarly, computing system 114 may receive one or more compressed embedding tables 164 packaged together with compression model 154.

[0087] At 404, the computing system receives input for processing based on an embedding table that includes individually compressed embeddings. In the example, a compressed model 152 running on computing system 112 receives input for processing using one or more compressed embedding tables 162. For example, compressed model 152 may be a neural network model that processes the input using compressed embedding tables 162. In one example, compressed model 152 may be targeted at one or more aspects of natural language processing (NLP), such as language classification, speech recognition, speech segmentation, etc. Furthermore, each of the one or more compressed embedding tables 162 may store word embeddings associated with words in a specific language. For example, different compressed embedding tables 162 may each be used to store word embeddings from various different languages.

[0088] At 406, the computational system determines a lookup value based on the input to search for an index in the embedding table of the associated embedding. In the example, compression model 152 uses the input to determine the lookup value used to find the embedding in one or more compressed embedding tables 162. In some examples, compression model 152 analyzes the input and extracts one or more features from the input to use as lookup values ​​to locate the embedding in one or more compressed embedding tables 162.

[0089] In the example, compression model 152 can analyze the word as input and extract one or more lookup values ​​based on that word. For example, compression model 152 can determine and extract the root of the word to use as a lookup value. In another example, compression model 152 can determine and extract one or more unary tuples to use as lookup values ​​based on the word input. Similarly, compression model 152 can determine and extract one or more bigrams, triples, and / or one or more other value groups based on any embedding feature to use as lookup values ​​based on the word input. Although the examples above mention words as example input, any type of data that generally represents any kind of information or content (e.g., audio, visual, text, etc.) can be used to determine lookup values ​​for locating embeddings in compressed embedding table 162 based on any one or more embedding features.

[0090] At 408, the computation system locates the individually compressed embedding in the embedding table based on the determined lookup value. In the example, compression model 152 uses, for example, a determined lookup value from the input at 406 to locate the embedding in the associated compressed embedding table 162. For example, compression model 152 may search the index using the determined lookup value or some variation of the determined lookup value based on the mapping associated with the index of compressed embedding table 162 to find the corresponding embedding in compressed embedding table 162.

[0091] In the example, compression model 152 uses one or more hashes and / or other operations associated with an index of compressed embedding table 162 to process a determined lookup value. For example, the resulting index of embedding table 162 may be associated with one or more hashes and / or other operations used to map input data to embedding table index values. In some examples, this mapping may typically map different possible input values ​​to the same index in compressed embedding table 152 in several instances, thus reducing the total number of entries and decreasing the size of compressed embedding table 152. In some examples, one or more hashes and / or other operations may include one or more data processing operations that may be performed before and / or after any particular hash.

[0092] In the example, compression model 152 performs one or more hash operations on the determined lookup value to search for an index in compressed embedding table 162. Compression model 152 then performs a lookup operation to search for an index in compressed embedding table 162 using the result of the one or more hash operations on the determined lookup value. The compression model then locates one of the indexes in compressed embedding table 162 that matches the result of the hash operations to obtain the associated embedding.

[0093] At 410, the computation system decompresses individually compressed embeddings independently of any other embeddings in the embedding table. In the example, compression model 152 locates one of the indices in compressed embedding table 162, where a specific index corresponds to one of the individually compressed embeddings in compressed embedding table 162. Compression model 152 decompresses a specific embedding individually, independently of any other embeddings in compressed embedding table 162. In various examples, the compression model decompresses the corresponding entry in compressed embedding table 162 independently of the underlying machine learning platform, for example, based on compression that is not available from the machine learning platform used to run compression model 152 or is otherwise not supported by the machine learning platform. In some examples, compression model 152 may be configured to perform the lookup and individual decompression of individually compressed embeddings as part of a single operation. Furthermore, compression model 152 may be configured such that the lookup and individual decompression of individually compressed embeddings are self-contained and invisible to the underlying machine learning platform or its users.

[0094] Figure 5 A flowchart depicts an example method for providing efficient embedded table storage and lookup for machine learning models according to an example embodiment of this disclosure. Although Figure 5 The steps performed in a specific order are described for illustrative and discussion purposes only; however, the method disclosed herein is not limited to the specifically described order or arrangement. Without departing from the scope of this disclosure, the individual steps of method 500 can be omitted, rearranged, combined, and / or adjusted in various ways.

[0095] At 502, the computing system receives one or more embedding tables comprising individually compressed rows. In one example, the computing system (e.g., computing system 112) receives one or more compressed embedding tables 162 packaged together with a compression model 152. In various examples, each compressed embedding table 162 stores individually compressed embeddings, each individually compressed embedding associated with one of the indices of the corresponding compressed embedding table 162. Furthermore, each of the individually compressed embeddings can be decompressed independently of any other embeddings in the compressed embedding table and without requiring decompression of any other embeddings in the compressed embedding table.

[0096] In some examples, each of one or more compressed embedding tables 162 is based on a specific embedding feature of word embeddings. For example, a collection of one or more compressed embedding tables 162 can be associated with n-tuple embedding features. For instance, one of the compressed embedding tables 162 can store the index and embedding associated with word untuples. A second compressed embedding table 162 can store the index and embedding associated with word doublets. Furthermore, a third compressed embedding table 162 can store the index and embedding associated with word triplets. Although untuple, doublet, and triplet embedding features are generally described in the examples above, compressed embedding tables 162 can be used for any type of embedding feature associated with any type of data, and are not limited to text data or embedding features of text data.

[0097] At 504, the computing system receives input for processing according to one or more embedding tables. In the example, a compression model 152 running on computing system 112 receives input for processing using one or more compression embedding tables 162. For example, the input may include any type of data associated with audio, visual, text, and / or any other type of content. In some examples, the input may be words or phrases processed by the compression model 152 associated with natural language processing (NLP). Furthermore, each of the one or more compression embedding tables 162 may store word embeddings used by the compression model 152 when processing the input.

[0098] In some examples, the compression model 152 may receive one or more inputs, such as words in a phrase or sentence, and process each word as input. For example, each word in a sentence may be processed individually using one or more compressed embedding tables 162. For example, a single word, such as "hello," may be processed as input using a untuple-based compressed embedding table 162, a biguple-based compressed embedding table 162, and / or a triple-based compressed embedding table 162 (and / or any other type of embedding feature). In some examples, groups of words that include one or more words may also be processed as input.

[0099] In 506, the computational system extracts a set of one or more lookup values ​​based on the input. In the example, compression model 152 analyzes the input and extracts one or more embedding features from the input to be used as lookup values ​​for locating embeddings in compressed embedding table 162. In one example, compression model 152 may receive a word such as “hello” and extract embedding features such as unary tuples of “hello” (e.g., 'h', 'e', ​​'l', 'l', 'o'), bigary tuples of “hello” (e.g., 'he', 'el', 'll', 'lo'), triplet tuples of “hello” (e.g., 'hel', 'ell', 'llo'), etc., to be processed using the corresponding compressed embedding table 162 associated with each specific embedding feature. For example, the unary tuple of “hello” can be processed using the compressed embedding table 162 of unary tuples, the bigary tuple of “hello” can be processed using another compressed embedding table of bigary tuples, and so on. Furthermore, the compression model 152 can typically process any number of inputs (e.g., tokens, entities, objects) based on any number of embedding features corresponding to at least one compressed embedding table 162. Additionally, it can typically process any type of input based on any one or more embedding features.

[0100] At 508, the computation system determines the embedding table index for each of the lookup values. In the example, compression model 152 uses one or more hashes and / or other operations to process each lookup value to determine the embedding table index to locate it in compressed embedding table 162. For example, compression model 152 can process each specific tuple value determined for "hello" by applying one or more hashes and / or other operations to each tuple value, where one or more hashes and / or other operations map the tuple value to the corresponding embedding table index value in compressed embedding table 162. Compression model 152 can then use the determined embedding table index to locate the associated tuple embedding in compressed embedding table 162. In other examples, tuples, triples, and / or any other embedding features can be similarly processed to obtain one or more embedding table indices for the corresponding compressed embedding table 162.

[0101] In 510, the computation system obtains the embedding of each determined embedding table index by decompressing only the specific corresponding row of the corresponding embedding table. In the example, compression model 152 uses the embedding table index value determined according to the lookup value to search for the index of compressed embedding table 162. For example, compression model 152 locates one of the indexes in the compressed embedding table that matches the embedding table index value determined according to the lookup value, and decompresses the specific embedding of compressed embedding table 162 associated with the located index separately. Specifically, compression model 152 can decompress a specific individually compressed embedding of compressed embedding table 162 independently and without needing to decompress any other embeddings of compressed embedding table 162.

[0102] In some examples, compression model 152 locates and decompresses one or more embeddings individually based on the embedding features associated with the input. For example, compression model 152 can determine an embedding table index as input for each unpaired embedding feature value associated with the word "hello". Compression model 152 can then search for the corresponding unpaired compressed embedding table 162 for each in the embedding table index and decompress each of the located embeddings individually based on the matching compressed embedding table 162 index. The compression model can then process the individually decompressed embeddings from the compressed embedding table 162.

[0103] In some examples, compression model 152 can locate and decompress embeddings individually from other compressed embedding tables 162 associated with other embedding features. For example, compression model 152 can also determine an embedding table index for each tuple embedding feature value associated with the word "hello", search for different corresponding tuple compressed embedding tables 162 for each of those embedding table indices, decompress each of the located embeddings individually based on the matching compressed embedding table 162 index, and process the decompressed embeddings accordingly. In various examples, compression model 152 can perform similar operations to locate and decompress embeddings individually from any compressed embedding table 162 corresponding to any embedding feature of any type of input.

[0104] At 512, the computational system processes decompressed embedding data obtained from one or more compressed embedding tables. In the example, compression model 152 processes one or more groups of individually decompressed embedding data from one or more compressed embedding tables 162. In various examples, compression model 152 uses the decompressed embedding data to run a model and generate output. In some examples, compression model 152 averages multiple distinct individually decompressed embeddings obtained from compressed embedding table 162. In one example, compression model 152 may average multiple decompressed embeddings, such multiple distinct numerical vectors being obtained for each of multiple distinct lookup values ​​associated with an embedding feature. For example, compression model 152 may average or compute the embedding mean for numerical vectors obtained for lookup values ​​associated with a unpaired tuple, a double tuple, a triple tuple as an embedding feature, and / or based on any type of input for any embedding feature. In some examples, compression model 152 may concatenate multiple decompressed embeddings, which may be further processed and / or provided as output for a specific operation or layer. In some examples, the compressed embedding model 152 can process such data by averaging, concatenating, and / or generally processing individually decompressed embedding data from different embedding features in any way. Furthermore, the compressed model 152 can process any number of individually decompressed embeddings in any way to generate any type of output.

[0105] At 514, the computational system generates output based on processing the decompressed embedding data obtained from each corresponding embedding table. In the example, compression model 152 can provide decompressed embedding data as output. In one example, the decompressed embedding data can be provided as an average or calculated average of the decompressed embedding data obtained from one or more compressed embedding tables 162. In one example, such decompressed embedding data can be provided as combined or concatenated data. In some examples, compression model 152 provides prediction, classification, determination, and / or other results based on analyzing and processing the decompressed embedding data obtained from one or more compressed embedding tables 162.

[0106] Example devices and systems

[0107] Figure 6A A block diagram of an example computing system 600 providing efficient embedded table storage and lookup in a machine learning model according to an example embodiment of the present disclosure is depicted. The example computing system 600 includes a user computing device 602, a server computing system 630, and a training computing system 650 communicatively coupled via a network 680.

[0108] User computing device 602 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0109] User computing device 602 includes one or more processors 612 and memory 614. The one or more processors 612 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 614 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 614 can store data 616 and instructions 618, which are executed by processor 612 to cause user computing device 602 to perform operations.

[0110] In some examples, the user computing device 602 can store or include one or more machine learning models 620. For example, the machine learning model 620 can be, or can otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some example machine learning models can fully utilize attention mechanisms, such as self-attention. For example, some example machine learning models can include multi-head self-attention models (e.g., transformer models). In some examples, the machine learning model includes and uses one or more embedding tables (not shown).

[0111] In some implementations, one or more machine learning models 620 can be received from a server computing system 630 via a network 680, stored in memory 614, and then used or otherwise implemented by one or more processors 612. In some implementations, a user computing device 602 can implement multiple parallel instances of a single machine learning model 620.

[0112] Additionally or alternatively, one or more machine learning models 640 can be included in or otherwise stored and implemented by a server computing system 630 that communicates with the user computing device 602 according to a client-server relationship. For example, the machine learning model 640 can be implemented by the server computing system 630 as part of a web service. Thus, it is possible to store and implement one or more machine learning models 620 and any associated embedding tables (not shown) at the user computing device 602, and / or it is possible to store and implement one or more machine learning models 640 and any associated embedding tables (not shown) at the server computing system 630.

[0113] User computing device 602 may also include one or more user input components 622 for receiving user input. For example, user input component 622 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component can be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other means through which a user can provide input.

[0114] Server computing system 630 includes one or more processors 632 and memory 634. The one or more processors 632 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 634 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 634 can store data 636 and instructions 638 that are executed by processor 632 to cause server computing system 630 to perform operations.

[0115] In some implementations, the server computing system 630 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. In instances where the server computing system 630 includes multiple server computing devices, such server computing devices are capable of operating according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0116] As described above, the server computing system 630 is capable of storing or otherwise including one or more machine learning models 640. For example, the machine learning model 640 can be, or can otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models are capable of fully utilizing attention mechanisms, such as self-attention. For example, some example machine learning models are capable of including multi-head self-attention models (e.g., transformer models). In some examples, the machine learning model includes and uses one or more embedding tables (not shown).

[0117] User computing device 602 and / or server computing system 630 are capable of training machine learning models 620 and / or 640, and / or any embedding tables (not shown) associated with machine learning models 620, 640, via interaction with training computing system 650 communicatively coupled over network 680. Training computing system 650 may be separate from server computing system 630 or may be part of server computing system 630.

[0118] The training computing system 650 includes one or more processors 652 and memory 654. The one or more processors 652 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 654 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 654 can store data 656 and instructions 658 executed by the processor 652 to cause the training computing system 650 to perform operations. In some embodiments, the training computing system 650 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0119] The training computing system 650 can include a model trainer 660 that trains machine learning models 620 and / or 640 and / or any embedding tables (not shown) associated with machine learning models 620 and 640, stored at user computing device 602 and / or server computing system 630, using various training or learning techniques, such as, for example, error backpropagation. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update parameters over multiple training iterations.

[0120] In some implementations, backpropagation of execution errors can include truncated backpropagation through execution time. Model trainer 660 is capable of performing various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model. Furthermore, model trainer 660 is capable of training machine learning model 620 and / or machine learning model 640 based on a training data set (e.g., training data set 662), and / or any embedding tables (not shown) associated with machine learning models 620, 640.

[0121] In some implementations, training examples can be provided by the user computing device 602 if the user has consented. Therefore, in this implementation, the machine learning model 620 provided to the user computing device 602 can be trained by the training computing system 650 on user-specific data received from the user computing device 602. In some instances, this process can be referred to as model personalization.

[0122] Model trainer 660 includes computer logic for providing desired functionality. Model trainer 660 can be implemented in hardware, firmware, and / or software that controls the processor. For example, in some embodiments, model trainer 660 includes a program file that is stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 660 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or optical or magnetic media.

[0123] Network 680 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communication on Network 680 can be performed via any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0124] The examples provided in this disclosure can be used in association with machine learning models used in a variety of tasks, applications, and / or use cases. Furthermore, according to the examples in this disclosure, such machine learning models can leverage one or more embedded tables and achieve efficient embedded table storage and lookup.

[0125] In some embodiments, the input to the machine learning model of this disclosure can be image data. The machine learning model can process the image data to generate output. For example, the machine learning model can process image data to generate image recognition output (e.g., image data identification, latent embedding of image data, encoded representation of image data, hashing of image data, etc.). As another example, the machine learning model can process image data to generate image segmentation output. As another example, the machine learning model can process image data to generate image classification output. As another example, the machine learning model can process image data to generate image data modification output (e.g., image data alteration, etc.). As another example, the machine learning model can process image data to generate encoded image data output (e.g., encoded and / or compressed representation of image data, etc.). As another example, the machine learning model can process image data to generate scaled-up image data output. As another example, the machine learning model can process image data to generate prediction output.

[0126] In some implementations, the input to the machine learning model of this disclosure can be text or natural language data. The machine learning model can process the text or natural language data to generate output. As an example, the machine learning model can process natural language data to generate language-encoded output. As another example, the machine learning model can process text or natural language data to generate latent text embedding output. As another example, the machine learning model can process text or natural language data to generate translation output. As another example, the machine learning model can process text or natural language data to generate classification output. As another example, the machine learning model can process text or natural language data to generate text segmentation output. As another example, the machine learning model can process text or natural language data to generate semantic intent output. As another example, the machine learning model can process text or natural language data to generate scaled-up text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model can process text or natural language data to generate predictive output.

[0127] In some embodiments, the input to the machine learning model of this disclosure can be speech data. The machine learning model can process the speech data to generate output. As an example, the machine learning model can process speech data to generate speech recognition output. As another example, the machine learning model can process speech data to generate speech translation output. As another example, the machine learning model can process speech data to generate latent embedding output. As another example, the machine learning model can process speech data to generate encoded speech output (e.g., encoded and / or compressed representations of speech data, etc.). As another example, the machine learning model can process speech data to generate scaled-up speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model can process speech data to generate text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model can process speech data to generate predictive output.

[0128] In some implementations, the input to the machine learning model of this disclosure can be latent encoded data (e.g., a latent spatial representation of the input). The machine learning model can process the latent encoded data to generate an output. As an example, the machine learning model can process the latent encoded data to generate an identification output. As another example, the machine learning model can process the latent encoded data to generate a reconstruction output. As another example, the machine learning model can process the latent encoded data to generate a search output. As another example, the machine learning model can process the latent encoded data to generate a re-clustering output. As another example, the machine learning model can process the latent encoded data to generate a prediction output.

[0129] In some implementations, the input to the machine learning model of this disclosure can be statistical data. The machine learning model can process the statistical data to generate output. For example, the machine learning model can process the statistical data to generate an identification output. As another example, the machine learning model can process the statistical data to generate a prediction output. As another example, the machine learning model can process the statistical data to generate a classification output. As another example, the machine learning model can process the statistical data to generate a segmentation output. As another example, the machine learning model can process the statistical data to generate a visualization output. As another example, the machine learning model can process the statistical data to generate a diagnostic output.

[0130] In some embodiments, the input to the machine learning model of this disclosure can be sensor data. The machine learning model can process the sensor data to generate output. As an example, the machine learning model can process sensor data to generate identification output. As another example, the machine learning model can process sensor data to generate prediction output. As another example, the machine learning model can process sensor data to generate classification output. As another example, the machine learning model can process sensor data to generate segmentation output. As another example, the machine learning model can process sensor data to generate visualization output. As another example, the machine learning model can process sensor data to generate diagnostic output. As another example, the machine learning model can process sensor data to generate detection output.

[0131] In some cases, machine learning models can be configured to perform tasks that include encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task could be an audio compression task. The input could include audio data and the output could include compressed audio data. In another example, the input includes visual data (e.g., one or more images and / or videos), the output includes compressed visual data, and the task is a visual data compression task. In yet another example, the task could include generating embeddings for input data (e.g., input audio or visual data).

[0132] In some cases, the input includes visual data and the task is a computer vision task. In other cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, an image processing task could be image classification, where the output is a grouping, with each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to that object class. An image processing task could be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the likelihood that that region depicts an object of interest. As another example, an image processing task could be image segmentation, where the image processing output defines a corresponding likelihood for each class in a predetermined group of categories for each pixel in one or more images. For example, the group of categories could be foreground and background. As another example, the group of categories could be object classes. As another example, an image processing task could be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, an image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at pixels between the images in the network input for each pixel of one of the input images.

[0133] In some cases, the input includes audio data representing spoken utterance and the task is speech recognition. The output may include text output mapped to spoken utterance. In some cases, the task involves encrypting or decrypting the input data. In some cases, the task involves microprocessor performance tasks such as branch prediction or memory address translation.

[0134] Figure 6A The illustration shows an example computing system that can be used to implement this disclosure. Other computing systems can also be used. For example, in some embodiments, user computing device 602 can include a model trainer 660 and a training data set 662. In this embodiment, machine learning model 620 can be trained and used locally at user computing device 602. In some such embodiments, user computing device 602 can implement model trainer 660 to personalize machine learning model 620 based on user-specific data.

[0135] Figure 6B A block diagram depicts an example computing device 682 performing operations according to an exemplary embodiment of the present disclosure. The computing device 682 can be a user computing device or a server computing device.

[0136] Computing device 682 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application can include a machine learning model that may include and use one or more embedding tables. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0137] like Figure 6B As illustrated, each application is capable of communicating with multiple other components of the computing device, such as, for example, one or more sensors, a field manager, a device status component, and / or additional components. In some implementations, each application is capable of communicating with each device component using an application programming interface (API) (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0138] Figure 6C A block diagram depicts an example computing device 690 implemented according to an example embodiment of the present disclosure. The computing device 690 can be a user computing device or a server computing device.

[0139] The computing device 690 includes multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application is able to communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0140] The central intelligence layer comprises multiple machine learning models. For example, such as... Figure 6C As illustrated, a corresponding machine learning model can be provided for each application and managed by a central intelligent layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligent layer can provide a single model for all applications. In some embodiments, the central intelligent layer is included within or otherwise implemented by the operating system of the computing device 690.

[0141] The central intelligence layer can communicate with the central device data layer. The central device data layer can be the central data repository of the computing device 690. For example... Figure 6C As illustrated, the central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, a field manager, a device status component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a proprietary API).

[0142] Additional Disclosure

[0143] The technologies discussed in this paper refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and divisions of tasks and functions between and within components. For example, the processes discussed in this paper can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate continuously or in parallel.

[0144] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation and not as a limitation of this disclosure. Those skilled in the art, upon gaining an understanding of the foregoing, will readily be able to generate modifications, variations, and equivalents to these embodiments. Therefore, this disclosure does not exclude modifications, variations, and / or additions to the subject matter that will be apparent to those of ordinary skill in the art. For example, features described or illustrated as part of one embodiment can be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A computer-implemented method for performing efficient embedded table storage and lookup in a machine learning model, comprising: An embedding table associated with a machine learning model is obtained by one or more processors, the embedding table comprising multiple embeddings correspondingly associated with corresponding indexes of the embedding table; Each specific embedding of the embedding table is individually compressed by the one or more processors to allow each corresponding embedding of the embedding table to be decompressed independently of any other embedding in the embedding table; and The embedding table, which includes individually compressed embeddings, is packaged together with the machine learning model by the one or more processors.

2. The computer-implemented method according to claim 1, further comprising: The one or more processors update the corresponding indexes of the embedded table based on hash operations.

3. The computer-implemented method according to claim 1, further comprising: The one or more processors provide the embedding table, which includes individually compressed rows, packaged together with the machine learning model, to one or more computing devices.

4. The computer-implemented method according to claim 1, wherein, The compression is performed independently of the machine learning platform.

5. The computer-implemented method according to claim 1, wherein, The compression is performed using compression that cannot be obtained from the machine learning platform.

6. The computer-implemented method according to claim 1, wherein, The corresponding embedding in the embedding table is associated with an item in one of the multiple items included in the vocabulary.

7. The computer-implemented method according to claim 1, wherein, The machine learning model is associated with the embedding layer of the neural network.

8. A computer-implemented method for performing efficient embedded table storage and lookup in a machine learning model, comprising: An embedding table associated with a machine learning model is obtained by one or more processors, the embedding table comprising multiple individually compressed embeddings to allow each corresponding embedding to be decompressed independently of any other embedding in the embedding table; The one or more processors receive input used to locate the embedding in the embedding table; The one or more processors determine a lookup value based on the input to locate the embedding by searching the index of the embedding table; The embedding is located by the one or more processors searching the index of the embedding table based on the determined lookup value; as well as The embedding is decompressed by the one or more processors based on the location, independently of any other embedding in the embedding table.

9. The computer-implemented method according to claim 8, further comprising: The decompressed embedding associated with running the machine learning model is processed by the one or more processors.

10. The computer-implemented method according to claim 8, wherein, The lookup value is determined based on a hash operation used to map the input to one of the indexes in the embedded table.

11. The computer-implemented method according to claim 8, wherein, The corresponding embedding in the embedding table is associated with an item in one of the multiple items included in the vocabulary.

12. The computer-implemented method according to claim 8, wherein, The decompression is performed independently of the machine learning platform.

13. The computer-implemented method according to claim 9, wherein, The decompression is performed using compression that cannot be obtained from the machine learning platform.

14. A computational system for performing efficient embedded table storage and lookup in machine learning models: One or more processors; and One or more non-transitory computer-readable media, said one or more non-transitory computer-readable media collectively storing: A machine learning model, configured to receive model input and process the model input to generate model output, wherein... The machine learning model includes an embedding table comprising individually compressed embeddings to allow each corresponding embedding to be decompressed independently of any other embeddings in the embedding table, and wherein the machine learning model is configured to perform operations including: Obtain the model input for processing based on the embedding table; The lookup value is determined based on the model input, and is used to search the index of the embedding table to locate the embedding; The embedding is located by searching the index of the embedding table for the determined lookup value; and The embedding is decompressed by the one or more processors based on the location, independently of any other embedding in the embedding table.

15. The computing system of claim 14, further comprising: The decompressed embeddings are processed using other corresponding decompressed embeddings from the embedding table.

16. The computing system according to claim 14, wherein, The operation further includes: The model output is generated based on the processing of the decompressed embedding.

17. The computing system according to claim 14, wherein, The lookup value is determined by a hash operation based on an index used to map the model input to one of the indexes in the embedded table.

18. The computing system according to claim 14, wherein, The corresponding embedding in the embedding table is associated with an item in one of the multiple items included in the vocabulary.

19. The computing system according to claim 14, wherein, The decompression is performed independently of the machine learning platform.

20. The computing system according to claim 14, wherein, The decompression is performed using compression that cannot be obtained from the machine learning platform.

Images