Acceleration of calculation of embedded layer

Abstract

To provide a method, a system, and an apparatus having a computer readable medium for executing neural network computing with using a system for executing a neural network on a hardware circuit.SOLUTION: A system includes a host for receiving a batch to be input to a neural network layer. Each of inputs is stored in a storage place identified by an address. The system identifies one or a plurality of overlapping addresses in an address list based on the one or the plurality of inputs. The system generates a unique identifier for use in identification of the overlapping addresses in the address list with respect to each of the overlapping addresses. Thus, the system (i) acquires a first input from the storage place identified by the address corresponding to the unique identifier and (ii) generates an output of a layer from the acquired first input.SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 909,667, filed October 2, 2019, which is incorporated herein by reference in its entirety.

[0002] background This specification relates generally to using circuits to perform neural network computations.

[0003] A neural network is a machine learning model that utilizes one or more layers of nodes to generate an output (e.g., a classification) for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to one or more other layers in the network (e.g., other hidden layers or the output layer of the network). Some of the layers of the network generate outputs from the received input according to the current values of their respective sets of parameters.

[0004] Some neural networks are convolutional neural networks (CNNs) (e.g., for image processing) or recurrent neural networks (RNNs) (e.g., for speech and language processing). Each of these neural networks includes a respective set of convolutional neural network layers or recurrent neural network layers. A neural network layer may have an associated set of kernels and an embedding layer to process inputs and generate a set of vectors for training the neural network. The kernels may be represented as tensors (i.e., multidimensional arrays) of weights. As an example, an embedding layer may process inputs (e.g., image pixel data inputs) or sets of activation values generated by a neural network layer. These sets of inputs or sets of activation values may also be represented as tensors. Summary of the Invention

[0005] overview This document describes a technique for accelerating the computation of an embedding layer of an artificial neural network. The technique can be used in a computing system (such as a large-scale distributed system) that includes circuitry configured to perform a deduplication operation on a list of addresses. The deduplication operation is performed to reduce load imbalance on the distributed system during processing of inputs in the embedding layer of the neural network (such as during an exemplary forward pass computation of the embedding layer).

[0006] Described herein are methods, systems, and apparatuses, including computer-readable media, for performing neural network computations using a system configured to run a neural network on hardware circuitry. The system includes a host that receives a batch of inputs to a neural network layer, each of the inputs being stored in a memory location identified by an address. The system identifies, for one or more inputs, one or more duplicate addresses in a list of addresses. For each duplicate address, the system generates a unique identifier that identifies the duplicate address in the list of addresses. The system (i) retrieves a first input from a memory location identified by an address corresponding to the unique identifier, and (ii) generates a unique identifier for the retrieved first input. The output of the above layer is generated from the input of

[0007] One aspect of the subject matter described herein can be embodied in a method for performing a neural network computation using a system configured to execute a neural network on a hardware circuit. The method includes: a host receiving a batch of inputs to a neural network layer, each input in the batch of inputs being stored in a memory location identified by an address; the method further includes identifying, for one or more inputs in the batch of inputs, one or more duplicate addresses in a list of addresses; generating, for each duplicate address, a unique identifier that identifies the duplicate address in the list of addresses; retrieving, for the batch of inputs, a first input from a memory location identified by an address corresponding to the unique identifier; and generating an embedded output for the neural network layer from the retrieved first input.

[0008] Each of these and other implementations may optionally include one or more of the following features: For example, in some implementations, the neural network layer is an embedding layer of the neural network, the embedding output includes an embedded feature vector, and the method further includes, by a sparse circuit of the system, determining an inverse mapping of the unique identifier to a duplicate address for a particular input in the batch of inputs to the embedding layer, and generating one or more embedded feature vectors as outputs of the embedding layer based on the inverse mapping of the unique identifier to the duplicate address for the particular input.

[0009] In some implementations, the method further includes generating a filtered address list for the batch of inputs based on one or more unique identifiers, and determining a partitioning of addresses among each processor of a plurality of processors for addresses included in the filtered address list, wherein obtaining the first input includes, for each partition of addresses, obtaining the first input from a memory location identified by an address in the partition of addresses using a respective processor assigned to retrieve an input from the partition of addresses.

[0010] In some implementations, generating the embedded output of the neural network layer based on the reverse mapping includes parsing the list of addresses, including the duplicate addresses, and reverse-mapping an embedded feature vector corresponding to a unique identifier generated for the duplicate address to the particular input within the batch of inputs to the neural network layer.

[0011] The method may further include the steps of: a requesting processor providing to a remote processor an address of a first input corresponding to a unique identifier and a request for the first input; the requesting processor receiving from the remote processor the first input stored in a memory location of a data shard assigned to the remote processor; and performing one or more reduction operations to generate the embedded output of the neural network layer from the first input obtained from the data shard.

[0012] In some implementations, the list of addresses is for a batch of inputs forming an input feature sample including a plurality of sets of input features, and generating the unique identifiers includes generating the unique identifiers for each duplicate address across each set of input features in the input feature sample. In some implementations, determining the reverse mapping of the unique identifiers to duplicate addresses includes: The step of determining the inverse mapping of the unique identifier to duplicate addresses for a particular input across multiple sets of the input characteristics.

[0013] In some examples, each partition of addresses is assigned to a data shard of activation or gain values corresponding to vector elements of a multidimensional sparse tensor, and each data shard of activation or gain values is assigned to a specific processor for multiplication by a corresponding weight vector to perform the neural network calculation. In some implementations, the overlapping addresses are specific addresses of memory locations that store inputs to a neural network layer, the specific addresses are common among corresponding data shards of each processor of the plurality of processors, and the specific addresses overlap among (i) the set of input features or (ii) samples of the input features.

[0014] Generating the embedded output of the neural network layer may include providing, from a remote processor to a requesting processor, a plurality of inputs stored across memory locations of a data shard assigned to the remote processor core, and performing a lookup to obtain a corresponding weight vector, the corresponding weight vector including a respective weight value for multiplying one of the plurality of inputs to generate a partial activation value for a subsequent neural network layer. In some implementations, some of the plurality of inputs are received from memory locations identified by non-duplicate addresses, or some of the plurality of inputs are received from memory locations in the original list identified by addresses in which no duplicate addresses exist.

[0015] Other implementations of this and other aspects include corresponding systems, devices, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or a combination thereof installed on the systems that, when operated, causes the systems to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the systems to perform the actions.

[0016] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages: A computing system includes circuitry usable to implement a processing scheme for accelerating an embedding layer process for multiple batches of inputs. The circuitry and processing scheme can be used to reduce or mitigate load imbalance on a multi-core processing unit (e.g., a distributed system of processors) by eliminating duplicate addresses in an address list used to look up inputs to an embedding layer of a neural network.

[0017] Having multiple duplicate addresses can be particularly problematic when computing or updating embeddings for neural networks. For example, samples in a batch of inputs may contain a variable number of features that need to be embedded, such as in the feature space of an embedding layer of a neural network. Each feature (or input) value has an embedding (e.g., a vector of values representing the neural network's trainable weights that are stored in the neural network's embedding table). While each value in a sample (or batch) has an embedding, different samples in the batch (and even different features within the same sample) may contain the same feature value. In some cases, even the same feature within a sample may contain the same feature value.

[0018] For example, when computing or updating embeddings, the same embedding table may be used for different features within a sample or batch, or for different samples or batches of input. Because the same input value may need to be fetched multiple times (e.g., from the same address in memory) when computing or updating an embedding, methods for eliminating duplicate occurrences of addresses can help reduce the computational load on distributed systems and improve or accelerate the computation of embedding layers of neural networks.

[0019] The techniques for deduplication can be performed at the source or requesting processor before the addresses in the list are broadcast to separate remote processors in a distributed system assigned to retrieve specific data elements of a given batch of inputs for processing in the embedding layer. The techniques described herein can be used to more efficiently compute embedding outputs for a neural network layer from the retrieved data elements for inputs in a batch. For example, the process of generating embedding outputs can include retrieving embeddings for one or more inputs in a batch and performing reductions on the embeddings for associated input features in the batch.

[0020] Compared to previous systems, the techniques described herein can be used to reduce or minimize the amount of computations performed to look up the embedding data, perform the reduction, and return a reduced vector of embedding values. In generating the embedding layer output, the techniques provide an improved way to balance the computational load between a requesting processor that processes reduced embedding values for a batch of inputs and a remote processor that looks up embeddings corresponding to the inputs in the batch.

[0021] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other possible features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram of an exemplary computing system. [Figure 2] FIG. 1 is a block diagram of an example circuit including an example multi-core processing unit. [Figure 3] FIG. 1 illustrates an exemplary host broadcasting a set of addresses to processors of an exemplary multi-core processing unit. [Figure 4A] FIG. 1 is a diagram associated with an example of a processing scheme including two or more processors of an exemplary multi-core processing unit. [Figure 4B] FIG. 1 is a diagram associated with an example of a processing scheme including two or more processors of an exemplary multi-core processing unit. [Figure 4C] FIG. 1 is a diagram associated with an example of a processing scheme including two or more processors of an exemplary multi-core processing unit. [Figure 4D] FIG. 1 is a diagram associated with an example of a processing scheme including two or more processors of an exemplary multi-core processing unit. [Figure 5]FIG. 1 illustrates an exemplary processor of an exemplary multi-core processing unit. [Figure 6] FIG. 1 is a flow diagram illustrating an example process for generating outputs of a neural network layer. DETAILED DESCRIPTION OF THE INVENTION

[0023] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description Described herein are techniques for accelerating embedding layer computations to mitigate processing delays that may occur due to processor load imbalances in a distributed computing system. For example, these delays may occur during processor computations to generate the output of an embedding layer of a multi-layer neural network. Specifically, certain processors in a distributed system may be accelerated. If a processor or core must perform a significant number of lookup and computation operations (e.g., reduction or concatenation of retrieved values) to generate the embedding, that particular processor may experience processing delays corresponding to the load imbalance.

[0024] This imbalance may be between a source processor sending a request to look up or retrieve data for an input value and a remote processor performing the lookup to retrieve the data. This data may be an embedding (e.g., weight) for an embedding table, and the remote processor may be tasked with processing a request to return data or weight values for this embedding. For example, in a forward pass computation operation of the embedding layer, the remote processor processes a request from the source that causes the remote processor to look up an address for retrieving data stored in memory and perform a computation using the retrieved data. In previous distributed architectures, the remote processor performed minimal de-duplication of addresses specified in the request.

[0025] For example, in previous systems, a remote processor received one or more duplicate addresses and performed a data lookup and associated reduction operation corresponding to the input for all received addresses, including the duplicate addresses. However, the architecture and processing schemes used by some previous systems are prone to load imbalance problems, particularly when the computational load on one or more remote processors is not balanced with the computational load on the source processor or is not balanced between two or more remote processors. These load imbalances can stall or substantially delay the computational operations to generate the output of the embedding layer.

[0026] Generally, an embedding layer of a neural network is used to embed features into a feature / embedding space corresponding to the embedding layer. Features can be attributes or characteristics shared by the independent units on which analysis or prediction is performed. For example, these independent units can be words in a vocabulary or image pixels that form part of items such as images and other documents. An algorithm for training the embeddings in the embedding layer can be executed by a neural network processor to map features to embedding vectors. In some implementations, the embeddings in the embedding table are trained together with other layers of the neural network in which the embeddings are used. This type of training is done by backpropagating gradients to update the embedding table.

[0027] In other implementations, embeddings may be learned separately from other layers of the neural network in which they are used (such as when the embeddings are pre-trained). For example, an algorithm may be used by a neural network processor to compute embeddings by processing information about individual input features to determine a mapping or arrangement of similar inputs to geometrically close embedding vectors in the embedding space. In some cases, the process of computing embeddings may represent a technique for feature learning or feature engineering that allows a system to automatically discover the representations needed for feature detection from raw input data.

[0028] In some implementations, a given "input" may have one or more features of one or more types, and the embedding layer generates a respective embedding for each of those types. For example, the input may be for a search query with several different feature types. These feature types may include characteristics of the user or user device (e.g., location, preferences, device type, etc.), query tokens, previously submitted queries, or other related types that may correspond to attributes of the search query. For any feature type that has more than one feature for a given input, the computing system , to retrieve individual embeddings for each of those features, and the system is operable to combine the retrieved embeddings to generate a final embedding for that feature type, for example by calculating an average of the embedding values.

[0029] Based on the above context, techniques are described herein for implementing a special-purpose hardware circuit configured to perform address de-duplication at a source processor before transmitting a list of addresses to a remote processor for lookup in a portion of memory allocated to the remote processor. The hardware circuit includes an identifier (ID) scatter circuit ("scatter circuit") configured to analyze the original address list and remove one or more "duplicate" addresses. The scatter circuit is configured to filter duplicate addresses before conveying addresses belonging to each remote processor.

[0030] For example, a system including hardware circuitry can generate instructions to cause a scattering circuit to convey addresses belonging to data shards assigned to a particular remote processor of the system. The system then transmits these addresses to the particular remote processor assigned to the data shard that stores embedding vectors corresponding to the inputs (features) stored at these addresses. These embedding vectors may be part of weight values or embeddings in an embedding table stored across the data shards. In some cases, the embedding vectors in the embedding table are partitionable among multiple remote processors (processor nodes or cores) of the computing system. The system host is operable to transmit a set of addresses or vector indexes to a subset of the remote processors to perform operations associated with searching for a vector of weights (e.g., an embedding vector) that maps to a particular feature stored at an address within this set. In some examples, the remote processor performs one or more lookup and calculation operations before providing or returning the weight vector to the system host.

[0031] The hardware circuitry includes a reverse mapper configured to enhance or modify addresses by adding metadata (such as a hash key) to the addresses. This metadata is used to reverse map a particular set of looked-up or calculated data (e.g., a collected vector) to its corresponding original address prior to filtering of duplicate addresses. In some cases, the circuitry may be configured to perform a sparse reduction operation on data received from a remote core to form a dense matrix.

[0032] For example, the circuitry may be modified to walk through each address in the original address location list before deduplication, collect vector outputs corresponding to particular input addresses using metadata tags generated by the reverse mapper, and perform a reduce operation on the collected vectors. These techniques enable computations in a distributed system of processors to be performed by a particular primary processor owning the processing of multiple batches of inputs, instead of by a remote processor that is mapped to a particular portion of memory that holds weight values (e.g., parameters) corresponding to one or more inputs in the batch of inputs.

[0033] 1 is a block diagram of an exemplary computing system 100 configured to perform embedding layer operations to search for data elements stored in a memory of the system 100 to generate a set of embeddings for training a neural network. In some cases, training the neural network may include determining one or more learning targets based on an output vector representing an embedding of the neural network. For example, the embedding output may include a vector of floats / parameters for a given output dimension (e.g., 64-dimensional, 256-dimensional, etc.). It may correspond to one or more output feature sets.

[0034] Embedding outputs are generated when a neural network of system 100 is trained to perform a particular computational function (e.g., image or speech recognition). In some implementations, training a neural network involves updating a set of embeddings previously stored in the neural network's embedding table (e.g., during a prior phase of training the neural network). In other words, the embeddings of a neural network may be pre-trained. For example, the embeddings of an embedding layer of a neural network may not be trained together with the neural network for which the embeddings are used. Thus, the techniques described herein can be used to update embeddings of a pre-trained neural network, providing improved efficiency over previous approaches (e.g., approaches that may have been used to pre-train embeddings).

[0035] Computing system 100 includes a host 102, a multi-core processing unit 104, and data shards 106a-106k, where k is an integer greater than 1. In general, host 102 may be a processing unit (such as a processor, multiple processors, or multiple processor cores). Accordingly, host 102 may include one or more processors and is operable to generate or process instructions for accessing a target dense matrix and send instructions 110 to multi-core processing unit 104 to generate the target dense matrix. As described in more detail below, performing the embedding layer operation may include transforming sparse elements from one or more matrices to generate the dense matrix.

[0036] The multi-core processing unit 104 accesses corresponding elements 108a-108n from one or more of the data shards 106a-106k, where n is an integer greater than 1. The multi-core processing unit 104 generates a target dense matrix 112 using the corresponding elements 108a-108n and provides the target dense matrix 112 to the host 102 for further processing. The elements 108a-108n may be two-dimensional matrices having different sizes, and the multi-core processing unit 104 may generate the target dense matrix 112 by converting each of the elements 108a-108n into a vector and concatenating the n vectors into a single vector.

[0037] Generally, in the context of embedding, the "sparse" information corresponding to a sparse element may be a one-hot vector identifying a feature value. For example, if there are five possible values for a given feature (e.g., A, B, C, D, E), the sparse vector would identify feature value "A" as (1, 0, 0, 0, 0), and the embedding layer would map (1, 0, 0, 0, 0) to a dense embedding vector for feature value "A." In some implementations, during training of the embedding layer to learn embeddings, elements 108a-108n may be weight values in an embedding table that are converted to vectors (e.g., embedding vectors for feature values "B" or "C"). These weight values may be converted using a neural network processor in multi-core processing unit 104 that executes a training algorithm to compute embeddings based at least on the mapping of features to embedding vectors.

[0038] The host 102 can process instructions to update a target dense matrix and send the updated dense matrix to the multi-core processing unit 104. For example, the target dense matrix may correspond to an embedding of a pre-trained neural network. Accordingly, the host 102 can process instructions to update the embedding of the pre-trained neural network to generate an updated dense matrix. For example, during subsequent iterations of training the neural network to update the embedding, the embedding may be updated by performing a backward pass, determining a new mapping of input features to embedding vectors, and generating an updated dense matrix based on this new mapping. In this implementation, the multi-core processing unit 104 is operable to convert the updated dense matrix into corresponding sparse elements and update one or more sparse elements (e.g., weights) stored in the data shards 106a-106k accordingly.

[0039] As described above, host 102 is configured to process instructions for execution within computing system 100. In some implementations, host 102 is configured to process target dense matrix 112 generated by multi-core processing unit 104. In some other implementations, host 102 may be configured to request multi-core processing unit 104 to generate target dense matrix 112, and another processing unit may be configured to process target dense matrix 112.

[0040] Each processor of the multi-core processing unit 104 is configured to retrieve data elements stored in the memory of the system 100. This memory may include multiple data shards 106a-106k that store data including elements 108a-108n. This data may include inputs, activation values, gain values, or weight values corresponding to parameters or kernels of a weight matrix structure. In some implementations, the data shards 106a-106k may be one or more volatile memory units. In some other implementations, the data shards 106a-106k may be one or more non-volatile memory units. The data shards 106a-106k may also be another form of computer-readable media (such as a device in a storage area network or other configuration). The data shards 106a-106k may be coupled to the multi-core processing unit 104 using electrical, optical, or wireless connections. In some implementations, the data shards 106a-106k may be part of a multi-core processing unit 104 and may be based on a processor-in-memory (PIM) architecture.

[0041] The multi-core processing unit 104 is configured to determine a dense matrix based on the sparse elements. In some implementations, the multi-core processing unit 104 may be configured to determine the location of the sparse elements based on the dense matrix. The multi-core processing unit 104 includes multiple interconnected processors or processor cores. For example, the multi-core processing unit 104 may be a distributed processing system including multiple interconnected processor cores. In general, the terms “processor” and “processor core” may be used interchangeably to describe individual interconnected processing resources of the multi-core processing unit 104.

[0042] FIG. 2 is a block diagram of an exemplary special-purpose hardware circuit 200 that includes a multi-core processing unit 104. In general, computing system 100 can be implemented using one or more of circuits 200 (described below). In the implementation of FIG. 2, multi-core processing unit 104 has a set of interconnected processors, including processors 0-15. Each of processors 0-15 of multi-core processing unit 104 can be assigned to one or more of data shards 106a-106k to retrieve data (e.g., feature values and weights) stored in memory locations of the data shards assigned to the processor, as described in more detail below. Each processor of multi-core processing unit 104 is operable to fetch or retrieve data from a memory location identified by an address received in a request from host 102 or another processor within multi-core processing unit 104.

[0043] These processors cooperate to look up data elements that map to an embedding table. These data elements are looked up to perform embedding layer operations (e.g., calculations to generate an output vector (e.g., a dense vector) that represents the embedding of the neural network). In some implementations, the embedding output is stored as an object (e.g., a vector in a vocabulary). An embedding is a mapping of individual inputs in a batch of inputs to a dense vector of numeric values (e.g., floating-point values) that can be used to define relationships between inputs (e.g., terms or words). For one input in a given batch, the embedding may be a vector, but for the entire batch of inputs, the embedding may be a matrix (e.g., a collection of vectors). In some implementations, the vector of embedding values corresponds to trainable weights learned by the neural network during training. The learned weights can encode one or more words by looking up dense vectors stored in an embedding table, where the dense vectors correspond to particular words. Multiple dense vectors may correspond to dense matrices used to train a neural network to perform a particular task or function, for example, related to recognizing sequences of words for language processing.

[0044] In some examples, the embedding layer computation is a large-scale data lookup problem commonly used during the training phase of a neural network. For example, during training, a large table of weight values may be sharded or partitioned across various dimensions of a multidimensional tensor (e.g., the x-, y-, or z-dimensions of the multidimensional tensor). Specifically, the table of weight values may be sharded across data shards 106a-106k such that each individual weight value is stored in a respective location in the memory that comprises the data shards.

[0045] Additionally, one or more of the data shards 106a-106k may be partitioned to specific processors 0-15 within the multi-core processing unit 104. Each shard or partition in the table corresponds to a data shard 106k assigned to a specific processor or processor core of the multi-core processing unit 104. In some examples, each processor core is a respective neural network processor. In other examples, each processor core is a respective core of a single neural network processor. In general, the computing system 100 may include multiple processors or multiple processor cores based on the desired configuration of the system.

[0046] 2 and hardware circuit 200, the circuit includes an identifier (ID) scatter circuit 202 (“scatter circuit 202”), which includes a hash generator 204 and a demapper 206. Each of hash generator 204 and demapper 206 is described below.

[0047] The scouring circuitry 202 is configured to process an original address list that may be received by the host 102. The original address list identifies storage locations within a memory of the system 100 (such as the memory containing the data shards 106a-106k). In FIG. 2, the scouring circuitry 202 is shown as being external to or separate from the host 102 and the multi-core processing unit 104. However, in some implementations, the scouring circuitry 202 may be included as part of the host 102 (e.g., as a scouring unit or computing module or sub-circuit of the host 102).

[0048] Similarly, in some implementations, the scouring circuitry 202 may be included in the multi-core processing unit 104 as part of a processor within the multi-core processing unit 104. For example, the scouring circuitry 202 may be included in a source / primary processor (e.g., processor 0) of the multi-core processing unit 104. This source processor may be tasked by the host 102 to generate one or more requests and provide these requests to multiple other processors (e.g., processors 1-15) within the multi-core processing unit 104 to retrieve data elements from storage locations of data shards 106a-106k assigned to those processors.

[0049] The scatter circuitry 202 communicates with the host 102 to receive the original address list. For example, the host 102 receives batches of inputs to the neural network layer, such as from a source external to the system 100 or a controller that manages the machine learning operations performed by the system. For example, the machine learning operations may be for training a neural network to process image or audio data so that the neural network can learn how to recognize particular objects in images / video or how to detect particular word sequences in an audio stream.

[0050] The host 102 receives one or more batches of inputs. In some implementations, the host 102 receives multiple batches of inputs corresponding to one or more samples. For example, a first set of the multiple batches of inputs may correspond to one sample 210-1, while a second set of the multiple batches of inputs may correspond to another sample 210-2. Each sample 210-1, 210-2 may include inputs corresponding to a set of features of an object (e.g., an image or an audio stream). For example, each of the samples 210-1 and 210-2 may include a respective batch of inputs for features corresponding to different regions of the same image or for features corresponding to different regions of multiple different images.

[0051] Each input in the batch of inputs is stored in a memory location, which may be identified by an address (e.g.,

[99] ,

[0736] ,

[37] ,

[99] , etc.) in the original address list received by the scattering circuitry 202 from the host 102. In some implementations, each input has a corresponding set of weights, which may be stored as data or sparse elements of an embedding table that is sharded across data shards 106a-106k and partitioned across processors 0-15. In some implementations, the embeddings of individual features in the batch of inputs are stored in a distributed manner (e.g., across various data shards of the system 100).

[0052] For example, these features may be distributed across multiple processor nodes or cores (0-15), with each node being assigned memory addresses where to store shards of the embedding. As described in more detail below, host 102 (e.g., a source processor) provides requests to one or more of processors 0-15 (e.g., remote processors) that include one or more subsets of addresses to retrieve data elements and process them to generate a dense matrix.

[0053] As noted above, in some previous systems, a remote processor receiving a duplicate address would perform data lookup, arithmetic operations (e.g., multiplication), and associated reduction operations on data obtained for the received address, including the duplicate address. This created load imbalance problems, especially when the computational load on the remote processor was not balanced with the computational load on the requesting or source processor.

[0054] The scatter circuit 202 is configured to analyze the original address list to remove one or more "duplicate" addresses. The address list received by the host 102 includes one or more duplicate addresses corresponding to input features in the batch. For example, the original address list received at the system 100 may include addresses

[26] ,

[96] ,

[99] , [1], [7],

[0312] ,

[99] , and

[0912] for the batch of input. In this example, the original address list includes address

[99] as a duplicate address in the list. The scatter circuit 202 is configured to convey addresses belonging to each remote processor and to filter the duplicate addresses before providing the remote processor with a request to retrieve a data element corresponding to the conveyed address.

[0055] The scattering circuit 202 scatters one or more addresses in the list of addresses for each input in the batch of inputs. identifies multiple overlapping addresses. For example, a single embedding table can be referenced by multiple features. Thus, there can be overlapping addresses across multiple different features. For example, embedding table T can be referenced by features F1 and F2. In this example, an address (e.g.,

[99] ) can be repeated within the input of F1 (in one or more samples) and within the input of F2 (in one or more samples).

[0056] The sparging circuit 202 is configured to perform de-duplication on the original addresses by removing duplicate addresses and providing unique addresses (identifiers) to the multi-core processing unit 104 for processing by one or more of the processors 0-15. For each duplicate address, the sparging circuit 202 generates an identifier that is used to uniquely identify the duplicate address in the original address list. The sparging circuit 202 then filters the original address list to generate a filtered address list based on the particular identifiers that indicate duplicate addresses in the original address list, such that the filtered address list includes only unique identifiers that correspond to non-duplicate addresses.

[0057] More specifically, for each address in the original list, the sparging circuit 202 uses the hash generator 204 to generate an identifier corresponding to a hash ID. The memory of the system 100 may have an address value (e.g.,

[99] ) for various memory offsets (e.g., offset 0, offset 1, etc.). The hash generator 204 may generate a hash ID based on this address value, the memory offset of this address value, or a combination of each. For example, a portion of the hash ID generated based on the address value (e.g.,

[99] ) will repeat due to duplicate occurrences of that address in the list, while a second portion of the hash ID generated based on the memory offset or a combination of the address value and the memory offset will be different. In some examples, the sparging circuit 202 may add each first portion of the hash ID to an address in the original list as a prefix of that address. The sparging circuit 202 may recursively scan the address list to identify repeated occurrences of the added prefix and remove that particular address based on the repeated occurrence.

[0058] In some implementations, the first portion of the added prefix or hash ID is an example of a metadata tag used by the sparging circuit 202 to enrich the addresses in the original list to generate a filtered address list. The sparging circuit 202 generates the filtered address list for a batch of input based on the unique identifiers generated by the hash generator 204. The filtered address list includes only unique identifiers or addresses (e.g., non-duplicate addresses) of storage locations in the data shards 106. The filtered address list is then provided to the multi-core processing unit 104 so that processors 0-15 receive only unique addresses rather than multiple occurrences of duplicate addresses.

[0059] The scatter circuit 202 uses a reverse mapper 206 to generate a reverse map and uses this reverse map to reverse map the collected vectors to specific addresses in the original list. The reverse mapper 206 is configured to add one or more metadata tags to addresses or batches of inputs to enable the reverse mapping. For example, the reverse mapper 206 can enhance or modify addresses in the original list by adding metadata to the addresses. This metadata can be a metadata tag including a hash ID or annotation that specifies the memory offset of the address. The metadata is used by the reverse mapper 206 to reverse map a specific set of searched or calculated data (e.g., collected vectors) to the corresponding original addresses before filtering duplicate addresses.

[0060] In this manner, circuit 200 provides special-purpose hardware operable to step through each address in the original address location list prior to de-duplication and collect vector outputs corresponding to particular input addresses using metadata tags generated by demapper 206. As described in more detail below, circuit 200 is operable to perform a reduction operation on the collected vectors to generate a dense matrix. For example, host 102 or an exemplary source processor of multi-core processing unit 104 may include exemplary sparse reduction circuitry configured to perform a reduction of data elements received from remote cores to form a dense matrix, as described in more detail below with reference to FIG. 5.

[0061] 3 is a diagram illustrating host 102 broadcasting a set of addresses to each of processors 0, 4, 8, and 12 of multi-core processing unit 104. As shown in FIG. 3, host 102 transmits unique identifiers or addresses for batches of inputs to the processors of multi-core processing unit 104. In some implementations, host 102 may transmit a unique identifier for each batch of inputs corresponding to sample 210-1 and each of sample 210-1. For example, host 102 may transmit addresses [0, 26, 96, 99] for the inputs of batch 302 to processor 12 and addresses [2, 9, 17, 736] for the inputs of batch 304 to processor 0. As shown in FIG. 2, the addresses for the inputs of batch 302 are for sample 210-1, while the addresses for the inputs of batch 304 are for sample 210-2.

[0062] In some implementations, the host 102 is operable to provide different batches of input to different processors and different columns of processors within the multi-core processing unit 104. For example, the host 102 can send different batches of input to different processors in a sequential manner or in parallel. Similarly, the host 102 can send different batches of input to different columns of processors in a sequential manner or in parallel.

[0063] 4A-4D each illustrate a respective diagram associated with an example of a processing scheme 402 that can be implemented in system 100 using at least host 102 and two or more processors of multi-core processing unit 104.

[0064] As described above, processors 0-15 cooperate to retrieve data elements that map to an embedding table (e.g., a sparse embedding table including multiple sparse elements). These data elements are retrieved in response to the remote processor processing a request provided by the source (or requesting) processor. In some cases, these data elements are retrieved to perform embedding layer operations (e.g., calculations to generate an output vector (e.g., a dense vector) representing the embedding of the neural network, or calculations to generate partial activation values) in response to multiplying inputs (e.g., gain values or activation values) by a weight vector to obtain partial activation values.

[0065] Also, as described above, the embedding table may be a table of weight values that is sharded across data shards 106a-106k such that individual weight values are stored in respective locations in memory. The data shards 106a-106k may be partitioned to particular processors 0-15 within the multi-core processing unit 104 such that the weight values in the embedding table correspond to the data shards assigned to particular processors 0-15 of the multi-core processing unit 104.

[0066] As described below (and with reference to FIG. 5), a request provided from a source processor to a remote processor to retrieve a data element may include an address (e.g.,

[0912] ) identifying a memory location of a data shard. This memory location stores data about an input (such as a gain value or activation value). The request is provided to cause the remote processor to retrieve a data element that represents an input (such as one of multiple inputs in a batch of inputs received at host 102). In some implementations, the request is provided to cause the remote processor to perform a lookup operation to retrieve a particular or unique weight that corresponds to a particular input in the batch of inputs.

[0067] 4A, block 402 illustrates an exemplary processing scheme in which at least processors 0-3 cooperate to search for data elements that map to portions of an embedded table. In this example, processor 0 may be the source (or requesting) processor that sends respective requests to each of processors 1, 2, and 3. More specifically, processor 0 is the source processor, and each of processors 1, 2, and 3 is a remote processor relative to processor 0.

[0068] In some cases, processors 0, 1, 2, and 3 form a subset 404. In this subset 404, each of processor 0, processor 1, processor 2, and processor 3 may be a source processor for other processors in subset 404, which are remote processors that receive and process requests from source processors in subset 404. In some cases, the source and remote processors in subset 404 may still receive and process requests from another processor in multi-core processing unit 104 or from host 102, which may be a primary source processor not included in the subset.

[0069] The exemplary scheme shown in block 402 may be part of a processing technique executed in system 100 to generate outputs of a neural network layer (such as the embedding outputs of an embedded neural network layer or the outputs of the first hidden layer of a neural network). The processing technique includes one or more remote processors providing data elements including multiple inputs and corresponding lookup values to a requesting processor. These data elements are stored across storage locations of data shards assigned to the remote processors.

[0070] In some implementations, the data elements are provided to the source processor in response to each of the remote processors performing a lookup to obtain a weight vector including respective weight values for multiplying a particular input of the plurality of inputs. For example, the remote processor may be allocated a portion of memory to hold kernel matrix weights that are multiplied by activation values stored at corresponding addresses in the address list received by the source processor. The inputs and weight values are multiplied to generate partial activation values, e.g., for a subsequent neural network layer (described below with reference to at least FIG. 4D). To further illustrate this processing technique, an exemplary operation will be described.

[0071] During an exemplary operation in system 100 for processing multiple batches of input, an original list of multiple addresses is received by host 102 or by another source processor (such as processor 0 of multi-core processing unit 104). Among the multiple addresses in the list may be address

[0912] . Address

[0912] may occur more than once in the received list for a given batch or sample. Specifically, address

[0912] may be one of multiple addresses that are duplicated within (i) a set of input features or (ii) a sample of input features. In some implementations, each duplicate occurrence of address

[0912] within a batch or sample may be one of multiple addresses that are duplicated within a given data shard. The memory offset may be for a specific memory offset.

[0072] Address 0912 may be a location in memory that stores input / feature values that are identifiers (e.g., one-hot identifiers) of words in a given vocabulary (such as the word "car"). For example, an embedding table may map one-hot identifiers of words to embeddings. More specifically, address 0912 may store instances of input features that occur multiple times within the feature values of a given sample and / or within a given batch of inputs. Each occurrence of address 0912 may be a feature with an embedding (e.g., a particular weight or vector of trainable weight values stored in an embedding table in memory).

[0073] The system 100 can perform de-duplication on the original list to remove multiple duplicate occurrences of address , generating a filtered address list containing only unique addresses. The inverse mapper 206 can attach metadata tags to specific occurrences of address within a batch or sample to map a collected vector (e.g., a vector of weights) back to that specific occurrence of address within the original list. As described in more detail below, the exemplary source processor can use this inverse mapping to locally perform a calculation (e.g., multiplication) between (i) an input value stored in this specific occurrence of address and (ii) a collected vector received from a remote processor.

[0074] In general, address 0912 may identify a memory location in a data shard assigned to processor 0 that stores a particular input value. In some implementations, address 0912 in the data shard assigned to processor 0 may also be repeated 100 times in each of multiple batches of input, but refer to various data shards assigned to other processors of processing unit 104. For example, address 0912 may be a particular address that is common among respective data shards assigned to one or more of the processors in multi-core processing unit 104.

[0075] An exemplary source processor (e.g., host 102 or processor 0) may perform 16 memory lookups, one for each processor in multi-core processing unit 104 of system 100, but does not perform calculations related to reduction or matrix multiplication. In the example of FIG. 4A (and FIG. 4C, described below), processor 3 may send to processor 0 only its respective input values (e.g., activation values or gains) and weight values corresponding to unique addresses that may be generated for multiple overlapping occurrences of address 1012 within a particular batch.

[0076] 4B, remote processor 104n is configured to locally reduce its own lookup values based on instructions from the requesting processor. In some implementations, each of the processors in multi-core processing unit 104 may represent a neural network processor chip assigned to look up and locally reduce data elements stored in a particular data shard 106a-106k. Each remote processor is configured to look up and locally reduce data elements in the embedded table based on instructions provided by host 102 or based on instructions received directly from a requesting / source processor in a subset that includes the respective remote processor.

[0077] For example, processor 3 may be each remote processor in a subset of processors that includes processor 0, processor 1, and processor 2. In this example, processor 0 may be a requesting processor in the subset that includes processor 3, and processor 3 is a remote processor with respect to at least processor 0. Thus, Processor 3 is configured to look up and locally reduce data elements (such as the unique weight values for the inputs stored at address

[0912] ) to generate an example weight vector [f2].

[0078] 4C, the source or requesting processor (e.g., processor 0) receives the locally reduced data elements at each remote processor and performs one or more operations on those data values. In some implementations, the requesting processor is configured to perform additional reduction operations on the received values. For example, if the input and corresponding lookup data stored at address

[0912] are retrieved and locally reduced by processor 3 to generate the value of vector [f2], system 100 is operable to determine a reduction of the input for the corresponding sample or batch of inputs to which the input is assigned.

[0079] For example, using requesting processor 0 of subset 404, system 100 may determine that input

[0912] must be reduced five times for sample 210-1 and two times for sample 210-2. Furthermore, requesting processor 0 is configured to perform a reduction on all data values received from each remote processor in subset 404. For example, requesting processor 0 is configured to locally reduce each of weight vector [f2], weight vector [g1], and weight vector [f1] received from remote processor 1, remote processor 2, and remote processor 3, respectively. In some implementations, requesting processor 0 is configured to look up its own data elements (e.g., the input's unique weight values stored at addresses [1], [7], [2], [9], and

[17] ) and locally reduce the lookup values across these features to generate exemplary weight vectors [f0] and [g0]. The requesting processor may perform a reduction over its own lookup values and the data values received from each of the remote processors in subset 404 .

[0080] 4C , each of subsets 406 and 408 may be used to perform a processing scheme similar or identical to processing scheme 402 described above. For example, in subset 406, processor 7 is configured to look up and locally reduce data elements (such as the input's unique weight values stored at address

[37] ) to generate exemplary weight vector [e1]. In subset 406, the source or requesting processor (e.g., processor 4) receives the data elements for the locally reduced vector [e1] at remote processor 7 and performs one or more operations on those data values. Processor 4 may then generate exemplary weight vectors [d0] and [e0] based on one or more concatenation or reduction operations performed across the embeddings located for the various input values within the batch or sample.

[0081] In some implementations, each of processors 1-15 may send to processor 0 its respective input values (e.g., activation values or gains) and weight values corresponding to the unique addresses generated for multiple overlapping occurrences of address

[0912] within a particular batch. In other implementations, each of processors 0-15 may be assigned a respective batch for processing, each performing a respective calculation for 100 iterations of address

[0912] within the respective batch.

[0082] The processing techniques described with reference to FIGS. 4A-4D allow computations in a distributed system of processors to be performed by a particular primary processor responsible for processing multiple batches of inputs, instead of by a remote processor that is mapped to a particular portion of memory that holds weight values (e.g., parameters) corresponding to one or more inputs in the batch of inputs. These techniques also lead to improved load balancing and scalability of the system 100.

[0083] 4D, processor 12 may represent a processing node of an exemplary distributed processing system of system 100. In some implementations, processor 12 may be a primary processor owning the processing of multiple batches of input. Processor 12 is configured to receive each of the vector outputs (e.g., dense vectors) reduced by each requesting processor 104n in multi-core processing unit 104. More specifically, processor 12 may receive vector outputs (representing neural network embeddings) corresponding to data elements of each output matrix reduced locally by each remote processor.

[0084] Processor 12 is configured to perform calculations to generate partial activation values in response to multiplying the input (e.g., gain values or activation values) by the weight values of the weight vector to obtain the partial activation values. For example, processor 12 is configured to generate one or more sets of partial activation values based on multiplication between the input and the weight values of the weight vectors [a] and [b].

[0085] 5 illustrates an exemplary processor 104n of the multi-core processing unit 104, where n is an integer greater than 1. For example, the processor 104n may be any one of the processors 0-15 described above with reference to FIG. 2. Generally, the processor 104n is configured to receive a request 542 from the node network 520 to fetch or search data (e.g., sparse elements) stored in one or more data shards 530.

[0086] For example, the source processor 502 sends a request 542 to a processor 104n in the node network 520 to generate a dense matrix using the sparse elements. The node network 520 may be a two-dimensional mesh network. In some implementations, the request 542 to fetch data includes instructions for converting the fetched sparse elements into a dense matrix. As described above, this request for converting the fetched sparse elements into a dense matrix may be performed during training of an embedding layer to learn an embedding. For example, the fetched elements may be weight values of an embedding table that are converted into an embedding vector based on at least a mapping of input features to an embedding vector. The source processor 502 may broadcast the request 542 to the processors 104n. The routing of the broadcasted request 542 may be similar to the description of FIG. 2 , in which the host 102 provides the request to one or more processors 104n. In some implementations, the source processor 502 may be the host 102 or another processor 104n included in the multi-core processing unit 104.

[0087] In general, processor 104n is configured to receive a request 542 to fetch a data element stored in one or more data shards 530 and determine whether processor 104n is assigned to access the address of a location in memory that stores the data indicated by request 542. In some implementations, processor 104n may determine whether processor 104n is assigned to access the data element indicated by request 542 by using a lookup table. For example, if the address of a particular requested data element (e.g., address

[99] at memory offset 1 of lookup table No. 5) is included in the lookup table, processor 104n may send a signal to an exemplary data retrieval unit of the processor to cause the data retrieval unit to retrieve the particular requested data element. In some implementations, processor 104n may transmit the received request 542 to a node network. The network 520 may be configured to broadcast to another processor 104n on the network 520.

[0088] In response to processing the request 542, the processor 104n is configured to retrieve one or more requested data elements from a particular data shard 530 assigned to the processor 104n. In some implementations, the processor 104n is a vector processor, a vector processing unit (VPU), an array processing unit, or any suitable data computing resource for processing arrays and matrices of vector elements. In some implementations, each of the processors 104n included in the multi-core processing unit 104 may be assigned to a particular data element, for example, of an embedded table, stored in the data shard 530 based on the address and corresponding memory location used to store the data element.

[0089] In some implementations, the source processor 502 may generate one or more requests 542 for data elements corresponding to multiple batches of input to be retrieved and processed using one or more of the processors 104n. In some cases, each batch of input may be associated with a respective request 542 for a subset of the requested data elements. Each processor 104n is configured to independently retrieve assigned data elements from its assigned data shard 530. In some implementations, the processor 104n causes the retrieved data to be forwarded to one or more units of the processor for further processing, as described in more detail below.

[0090] 5, the processor 104n includes a sparse reduction unit 506, a concatenation unit 508, and a compression / decompression unit 510. The processor 104n can transfer retrieved data elements to the sparse reduction unit 506. The sparse reduction unit 506 is configured to reduce the dimensionality of the retrieved data elements. For example, the processor 104n may generate vector elements having dimensions of 100×1. The sparse reduction unit 506 is operable to receive a fetched data element having dimensions of 100×k and generate a sparse reduced element 548 by reducing the dimensionality of the fetched data element 346 to 100×1 through a logical operation, an arithmetic operation, or a combination of both. The sparse reduction unit 506 is configured to output the sparse reduced element 548 to the concatenation unit 508.

[0091] The concatenation unit 508 is configured to reorder and concatenate the sparse reduced elements 548 to generate a concatenated element 550. For example, the processor 104n may be configured to access data elements (e.g., sparse elements No. 1 through No. 100 of database table No. 1) including inputs to a neural network layer and corresponding sets of weight values. The processor 104n is operable to return the retrieved sparse element No. 10 (e.g., the first weight value) to the sparse reduction unit 506 before returning the retrieved sparse element No. 5 (e.g., the second weight value). The concatenation unit 508 is configured to reorder the later received sparse element No. 5 to be ordered before the previously received sparse element No. 10, and concatenate the sparse elements No. 1 through No. 100 as the concatenated element 550.

[0092] The compression / decompression unit 510 is configured to compress the connected elements 550 to generate a dense matrix 552 for the node network 520. For example, the compression / decompression unit 510 is operable to compress zero values in the connected elements 550 to improve the bandwidth of the node network 520. In some cases, multiple zero values may be generated in response to an arithmetic operation between an input and a weight (such as a multiplication between an input and a weight value having a zero value, resulting in a matrix of connected elements 550 having multiple zeros as a result of the multiplication). The compression / decompression unit 510 can compress this matrix to generate an exemplary dense matrix 552 for the node network 520.

[0093] In some implementations, the compression / decompression unit 510 may decompress a received dense matrix. For example, a first processor 104n (e.g., processor 9 in FIG. 2) may receive a dense matrix from a neighboring second processor 104n (e.g., processor 10 in FIG. 2) via the node network 520. The neighboring second processor 104n may decompress the received dense matrix and concatenate the decompressed dense matrix with the concatenated element 550 to form an updated concatenated element, which may be compressed and then output to the node network 520.

[0094] 6 is a flow diagram illustrating an example process 600 for generating outputs of a neural network layer. Process 600 can be implemented or performed using the system 100 described above. The description of process 600 can refer to the computing resources described above of system 100. In some implementations, the steps or actions of process 600 are enabled by programmed firmware or software instructions executable by one or more processors of the devices and resources described herein.

[0095] Referring now to process 600, a host of system 100 receives a batch of inputs to a neural network layer (602). For example, host 102 may receive one or more batches of inputs for processing by multi-core processing unit 104. Each input in the one or more batches of inputs is stored in a memory location identified by an address. For at least one batch of inputs, circuitry of system 100 identifies one or more duplicate addresses in a list of addresses for each input in the batch of inputs (604). For example, sparging circuit 202 is operable to scan the list of addresses to identify one or more duplicate addresses in the list of addresses for each input in the batch of inputs.

[0096] For each identified duplicate address, the circuitry generates a unique identifier that identifies the duplicate address within the list of addresses (606). For example, the scattering circuitry 202 is operable to generate a respective unique identifier for each duplicate address to identify the particular address as a duplicate within the list of addresses. The list of addresses may be for a batch of inputs that form a sample of input features that includes multiple sets of input features. The circuitry of the system 100 is operable to generate a unique identifier for each duplicate address across each set of input features within the sample of input features.

[0097] The system 100 generates a filtered address list for the batch of inputs based on the one or more unique identifiers generated by the circuitry (608). For example, the host 102 interacts with the scattering circuitry 202 to receive a corresponding unique identifier for each duplicate address, and generates a filtered address list for the batch of inputs based on at least the one or more unique identifiers generated by the scattering circuitry 202.

[0098] For the batch of inputs, system 100 retrieves a first input from a storage location identified by an address corresponding to the unique identifier based on the filtered address list (610). For example, based on the filtered address list, system 100 can retrieve the first input from a storage location of a data shard identified by an address corresponding to the unique identifier using a particular processor assigned to the corresponding data shard.

[0099] To obtain the first input, the system 100, the requesting processor, A request to obtain the first input may be provided to the remote processor for processing by the processor. The request may include an address for the first input corresponding to the unique identifier. In some cases, the request and address may be sent separately, but may also be sent simultaneously or in parallel. In some cases, the request and address may be sent sequentially, i.e., one before the other.

[0100] The requesting processor receives the first input from the remote processor in response to the remote processor processing the request, the first input stored in a memory location of the data shard assigned to the remote processor. In some implementations, as described in more detail below, the requesting processor is operable to perform one or more reduction operations to generate outputs of the neural network layer from the first input obtained from the data shard.

[0101] System 100 may determine the partitioning of the addresses in the filtered list among each of the multiple processors in multi-core processing unit 104. For example, system 100 may determine that the data element associated with address

[26] is partitioned into a data shard assigned to processor 13, and that the data elements associated with addresses

[96] and

[99] are partitioned into a data shard assigned to processor 15, or that the data element associated with address

[0912] is partitioned into a data shard assigned to processor 3.

[0102] Thus, for each partition of addresses, the system can obtain a first input from a memory location identified by an address in the partition of addresses, using a respective processor assigned to retrieve the input and other corresponding data elements (e.g., weight values) from the data shard assigned to the respective processor. For example, each partition of addresses can be assigned a data shard of activation or gain values corresponding to vector elements of a multidimensional sparse tensor. In some implementations, each data shard of activation or gain values is assigned to a specific processor for multiplication by a corresponding weight vector to perform a neural network calculation.

[0103] The system 100 generates 612 a neural network layer output (e.g., an embedded output) from the obtained first input. In some implementations, the neural network layer is an embedding layer of a neural network, and the embedded output includes an embedded feature vector. The sparse circuit 202 is configured to determine an inverse mapping of unique identifiers to duplicate addresses for a particular input within a batch of inputs to the embedding layer. In some implementations, the inverse mapping of unique identifiers to duplicate addresses is determined for a particular input across multiple sets of input features.

[0104] The system 100 is operable to generate one or more vectors (e.g., embedded feature vectors) as outputs of the embedding layer based on an inverse mapping of the unique identifiers to the duplicate addresses for particular inputs. For example, generating an output (e.g., an embedded output) of the neural network layer based on the inverse mapping may include parsing a list of addresses that includes duplicate addresses and mapping the embedded feature vectors corresponding to the unique identifiers generated for the duplicate addresses back to particular inputs in the batch of inputs to the neural network layer.

[0105] In some examples, generating the output of the embedding layer includes generating one or more dense matrices in response to processing data associated with the sparse table of embedding values. The data associated with the table may be represented in the form of a matrix. The processors and computing resources of the system 100 may be used to manipulate the data based on linear algebra algorithms, including matrix multiplication. The matrix may be a one-dimensional vector or a multidimensional matrix. The matrix may be represented by a data structure (such as a database table or variable). The embedded table may be sharded, for example, in the row and column dimensions, and stored in a portion of memory mapped to a particular processor among the multiple processors forming the multi-core processing unit 104.

[0106] Embodiments of the subject matter and the functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed herein and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing apparatus.

[0107] Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode and transmit information to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0108] The term "computing system" encompasses all types of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may include special-purpose logic circuitry (e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit)). In addition to hardware, an apparatus may also include code that creates an execution environment for a subject computer program (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these).

[0109] Computer programs (which may also be referred to or described as programs, software, software applications, modules, software modules, scripts, or code) may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form (such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment).

[0110] A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple associated files (e.g., a file that stores one or more modules, subprograms, or portions of code). A computer program may be deployed to be executed on one computer, or may be accessed by communication over a communications network, whether located at one site or distributed across multiple sites. The software may be deployed to run on multiple interconnected computers.

[0111] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit)).

[0112] A computer suitable for executing a computer program may include or be based on, by way of example, a general-purpose or special-purpose microprocessor, or both, or other types of central processing units. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or is operatively coupled to receive, send, or transmit data to or from one or more such mass storage devices. However, a computer need not have such devices. Furthermore, a computer may be incorporated into another device (e.g., a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few).

[0113] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0114] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form (including acoustic, speech, or tactile input). The computer can also interact with the user by sending documents to and receiving documents from a device used by the user (e.g., by sending a web page to a web browser on the user's client device in response to receiving a request from the web browser).

[0115] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, for example, as a data server, or in a middleware. The system may be implemented in a computing system that includes a back-end component (e.g., an application server), a front-end component (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. These components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN") (e.g., the Internet).

[0116] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0117] While this specification contains many specific implementation details, these should not be construed as limiting the scope of the invention or the claims, but rather as describing features specific to particular embodiments of a particular invention. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operating in a particular combination and initially as such, one or more features within a described combination may, in some cases, be excluded from this combination, and a described combination may be directed to a subcombination or a variation of the subcombination.

[0118] Similarly, while operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in a sequential order, or as requiring all of the shown operations to be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged in multiple software products.

[0119] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method for performing neural network computations using a system configured to execute a neural network on hardware circuitry, comprising: a host receiving a batch of inputs to a neural network layer, each input in the batch of inputs being stored in a memory location identified by an address, the method further comprising: identifying, for one or more entries in the batch of entries, one or more duplicate addresses in a list of addresses; generating, for each duplicate address, a unique identifier that identifies the duplicate address within the list of addresses; For the batch of inputs, obtaining a first input from a memory location identified by an address corresponding to the unique identifier; generating an embedded output of the neural network layer from the obtained first input.

2. The neural network layer is an embedding layer of the neural network, and the embedding output comprises an embedded feature vector, and the method further comprises: a scatter circuit of the system determining an inverse mapping of the unique identifier to a duplicate address for a particular input within the batch of inputs to the embedding layer; and generating one or more embedded feature vectors as outputs of the embedding layer based on the inverse mapping of the unique identifiers to the duplicate addresses for the particular input.

3. generating a filtered address list for the batch of inputs based on one or more unique identifiers; and determining, for the addresses included in the filtered address list, a partitioning of the addresses among each of a plurality of processors; 3. The method of claim 2, wherein obtaining the first input comprises, for each partition of addresses, obtaining the first input from a memory location identified by an address within the partition of addresses using a respective processor assigned to retrieve input from the partition of addresses.

4. generating the embedding output of the neural network layer based on the inverse mapping, 3. The method of claim 2, comprising parsing the list of addresses, including the duplicate addresses, and mapping embedded feature vectors corresponding to unique identifiers generated for the duplicate addresses back to the particular inputs in the batch of inputs to the neural network layer.

5. the requesting processor providing to the remote processor an address of a first input corresponding to the unique identifier and a request for said first input; receiving, by the requesting processor, from the remote processor, the first input stored in a storage location of a data shard assigned to the remote processor; and performing one or more reduction operations to generate the embedding outputs of the neural network layer from first inputs obtained from the data shards.

6. The list of addresses is for a batch of inputs forming an input feature sample comprising a plurality of sets of input features, and the step of generating a unique identifier comprises: The method of claim 2 , comprising generating the unique identifier for each duplicate address across respective sets of input features in the sample of input features.

7. The step of determining the reverse mapping of the unique identifier to a duplicate address comprises: The method of claim 6 , comprising determining the inverse mapping of the unique identifier to duplicate addresses for a particular input across multiple sets of the input features.

8. The overlap address is a specific address of a memory location that stores the input to a neural network layer, the specific address is common among the corresponding data shards of each processor of the plurality of processors; The method of claim 6 , wherein the particular addresses overlap among (i) the set of input features or (ii) the samples of input features.

9. Each partition of an address is assigned to a data shard of activation or gain values corresponding to vector elements of a multidimensional sparse tensor, The method of claim 2 , wherein each data shard of activation or gain values is assigned to a specific processor for multiplication by a corresponding weight vector to perform the neural network calculation.

10. generating the embedding output of the neural network layer, providing a plurality of inputs from a remote processor to a requesting processor, the inputs being stored across storage locations of data shards assigned to the remote processor core; and performing a lookup to obtain a corresponding weight vector, the corresponding weight vector including a respective weight value for multiplying one of the plurality of inputs to generate a partial activation value for a subsequent neural network layer.

11. a portion of the plurality of inputs are received from memory locations identified by non-overlapping addresses; or 11. The method of claim 10, wherein some of the plurality of inputs are received from storage locations identified by addresses in the original list for which no duplicate addresses exist.

12. 1. A system configured to execute a neural network on a hardware circuit to perform neural network computations, comprising: one or more processing devices; and one or more non-transitory machine-readable storage devices for storing instructions executable by the one or more processing devices to cause the one or more processing devices to perform operations, the operations comprising: a host receiving a batch of inputs to a neural network layer, each input in the batch of inputs being stored in a memory location identified by an address, the operation further comprising: identifying, for one or more entries in the batch of entries, one or more duplicate addresses in a list of addresses; generating, for each duplicate address, a unique identifier that identifies the duplicate address within the list of addresses; For the batch of inputs, obtaining a first input from a memory location identified by an address corresponding to the unique identifier; generating an embedded output of the neural network layer from the obtained first input.

13. the neural network layer is an embedding layer of the neural network, the embedding output comprises an embedded feature vector, and the operation further comprises: a scatter circuit of the system determining an inverse mapping of the unique identifier to a duplicate address for a particular input within the batch of inputs to the embedding layer; and generating one or more embedded feature vectors as outputs of the embedding layer based on the inverse mapping of the unique identifiers to the duplicate addresses for the particular input.

14. generating a filtered address list for the batch of inputs based on one or more unique identifiers; and determining, for the addresses included in the filtered address list, a partitioning of the addresses among each of a plurality of processors; 14. The system of claim 13, wherein obtaining the first input comprises, for each partition of addresses, obtaining the first input from a memory location identified by an address within the partition of addresses using a respective processor assigned to retrieve input from the partition of addresses.

15. generating the embedding output of the neural network layer based on the inverse mapping, 14. The system of claim 13, further comprising: parsing the list of addresses, including the duplicate addresses, and mapping back an embedded feature vector generated for the duplicate address corresponding to a unique identifier to the particular input within the batch of inputs to the neural network layer.

16. the requesting processor providing to the remote processor an address of a first input corresponding to the unique identifier and a request for said first input; receiving, by the requesting processor, from the remote processor, the first input stored in a storage location of a data shard assigned to the remote processor; and performing one or more reduction operations to generate the embedding output of the neural network layer from a first input obtained from the data shard.

17. The list of addresses is for a batch of inputs forming an input feature sample comprising a plurality of sets of input features, and the step of generating a unique identifier comprises: The system of claim 13 , comprising generating the unique identifier for each duplicate address across respective sets of input features in the sample of input features.

18. The step of determining the reverse mapping of the unique identifier to a duplicate address comprises:

20. The system of claim 17, comprising determining the inverse mapping of the unique identifier to duplicate addresses for a particular input across multiple sets of the input features.

19. The overlap address is a specific address of a memory location that stores the input to a neural network layer, the specific address is common among the corresponding data shards of each processor of the plurality of processors; 20. The system of claim 17, wherein the particular addresses overlap among (i) the set of input features or (ii) the sample of input features.

20. Each partition of an address is assigned to a data shard of activation or gain values corresponding to vector elements of a multidimensional sparse tensor, 14. The system of claim 13, wherein each data shard of activation or gain values is assigned to a specific processor for multiplication by a corresponding weight vector to perform the neural network calculation.

21. generating the embedding output of the neural network layer, providing a plurality of inputs from a remote processor to a requesting processor, the inputs being stored across storage locations of data shards assigned to the remote processor core; and performing a lookup to obtain a corresponding weight vector, the corresponding weight vector including a respective weight value for multiplying one of the plurality of inputs to generate a partial activation value for a subsequent neural network layer.

22. one or more non-transitory machine-readable storage devices for use in executing a neural network on hardware circuitry to perform neural network computations, the non-transitory machine-readable storage devices storing instructions executable by one or more processing units to perform operations, the operations including: a host receiving a batch of inputs to a neural network layer, each input in the batch of inputs being stored in a memory location identified by an address, the operation further comprising: identifying, for one or more entries in the batch of entries, one or more duplicate addresses in a list of addresses; generating, for each duplicate address, a unique identifier that identifies the duplicate address within the list of addresses; For the batch of inputs, obtaining a first input from a memory location identified by an address corresponding to the unique identifier; and generating an embedded output of the neural network layer from the obtained first input.

Images