Accelerating embedded layer computation

Abstract

Methods, systems, devices, and computer-readable media are described for performing neural network computations using a system configured to implement a neural network on hardware circuitry. The system includes a host receiving a batch of inputs to a neural network layer. Each input is stored in a memory location identified by an address. The system identifies one or more duplicate addresses in a list of addresses for the one or more inputs. For each duplicate address, the system generates a unique identifier that identifies the duplicate address in the list of addresses. The system (i) obtains a first input from a memory location identified by an address corresponding to the unique identifier; and (ii) generates an output of the layer from the obtained first input.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application 62 / 909,667, filed October 2, 2019, which is incorporated herein by reference. Background Art

[0003] This specification generally relates to using circuits to perform neural network computations.

[0004] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, such as a classification, for a given input. In addition to an output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to one or more other layers in the network, such as other hidden layers or the output layer. Some layers of the network generate outputs from received inputs based on the current values ​​of their corresponding parameter sets.

[0005] Some neural networks are convolutional neural networks (CNNs) (e.g., for image processing) or recurrent neural networks (RNNs) (e.g., for speech and language processing). Each of these neural networks includes a set of corresponding convolutional or recurrent neural network layers. A neural network layer can have an associated set of kernels and an embedding layer for processing inputs to generate a set of vectors for training the neural network. The kernel can be represented as a tensor, i.e., a multidimensional array, of weights. For example, an embedding layer can process a set of inputs, such as inputs of image pixel data or activation values ​​generated by a neural network layer. The input set or activation value set can also be represented as a tensor. Summary of the Invention

[0006] This article describes techniques for accelerating computation of an embedding layer of an artificial neural network. The techniques can be used in a computing system, such as a large-scale distributed system, that includes circuitry configured to perform a deduplication operation on a list of addresses. During processing of inputs to the embedding layer of the neural network, such as during an example forward pass compute of the embedding layer, the deduplication operation is performed to reduce load imbalance on the distributed system.

[0007] This specification describes methods, systems, and apparatus, and computer-readable media, for performing neural network computations using a system configured to implement a neural network on hardware circuitry. The system includes a host computer that receives a batch of inputs to a neural network layer. Each input is stored in a memory location identified by an address. The system identifies one or more duplicate addresses in a list of addresses for one or more inputs. For each duplicate address, the system generates a unique identifier that identifies the duplicate address in the list of addresses. The system (i) obtains a first input from a memory location identified by an address corresponding to the unique identifier; and (ii) generates an output of the layer from the obtained first input.

[0008] One aspect of the subject matter described in this specification can be embodied as a method for performing neural network computations using a system configured to implement a neural network in hardware circuitry. The method includes: receiving, by a host, a batch of inputs to a neural network layer, wherein each input in the batch of inputs is stored in a memory location identified by an address; identifying one or more duplicate addresses in a list of addresses for one or more inputs in the batch of inputs. For each duplicate address, the method includes: generating a unique identifier that identifies the duplicate address in the list of addresses; obtaining a first input for the batch of inputs from a memory location identified by an address corresponding to the unique identifier; and generating an embedding output for the neural network layer from the obtained first input.

[0009] Each of these and other embodiments can optionally include one or more of the following features. For example, in some embodiments, the neural network layer is an embedding layer of the neural network, the embedding output comprises an embedding feature vector, and the method further comprises decentralized circuitry of the system determining a reverse mapping of the unique identifier to a repetition address of a particular input in the batch of inputs to the embedding layer; and generating one or more embedding vectors as output of the embedding layer based on the reverse mapping of the unique identifier to the repetition address of the particular input.

[0010] In some embodiments, the method includes: generating a filtered address list of the batch input based on one or more unique identifiers; determining an address partition between each of a plurality of processors for addresses in the filtered address list; and wherein obtaining the first input includes: for each address partition, using a corresponding processor assigned to retrieve input from the address partition, obtaining the first input from a memory location identified by an address in the address partition.

[0011] In some implementations, generating the embedding output of the neural network layer based on the reverse mapping includes parsing the address list including the duplicate addresses to map the embedding feature vector corresponding to the unique identifier generated for the duplicate address back to the specific input in the batch of inputs to the neural network layer.

[0012] The method also includes: a request processor providing an address of a first input corresponding to a unique identifier and a request for the first input to a remote processor; the request processor receiving the first input from the remote processor, the first input being stored in a memory location of a data shard assigned to the remote processor; and performing one or more reduction operations to generate the embedding output of the neural network layer from the first input obtained from the data shard.

[0013] In some embodiments, the address list is used to form a batch input of input feature samples comprising multiple sets of input features, and generating the unique identifier includes generating the unique identifier for each repeated address on each corresponding set of input features in the input feature samples. In some embodiments, determining the reverse mapping between the unique identifier and the repeated address includes determining the reverse mapping between the unique identifier and the repeated address for a specific input of the multiple sets of input features.

[0014] In some examples, each address partition is assigned to a data slice corresponding to an activation value or gain value of a vector element of a multidimensional sparse tensor; and each data slice of the activation or gain values ​​is assigned to a specific processor for multiplication with a corresponding weight vector to perform the neural network calculation. In some embodiments, the repeated address is a specific address of a memory location storing an input to a neural network layer, the specific address is common between corresponding data slices of each processor in the plurality of processors, and the specific address is repeated between (i) a set of input feature sets, or (ii) the input feature samples.

[0015] Generating the embedded output of the neural network layer includes: providing a plurality of inputs from a remote processor to a requesting processor, the plurality of inputs being stored in memory locations of a data shard assigned to the remote processor core; and performing a lookup to obtain a corresponding weight vector, the weight vector including corresponding weight values ​​for multiplying inputs from the plurality of inputs to generate a partial activation for a subsequent neural network layer. In some embodiments, a portion of the plurality of inputs is received from the memory location identified by an address in the original list without duplicate addresses, or from a memory location identified by an address in the original list without duplicate addresses.

[0016] Other embodiments and aspects of the present invention include corresponding systems, devices, and computer programs configured to perform the methods encoded on computer storage devices. The system of one or more computers can be configured by means of software, firmware, hardware, or a combination thereof installed on the system so that, when operated, the system performs the actions. The one or more computer programs can be configured by having instructions that, when executed by a data processing device, cause the device to perform the actions.

[0017] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages. A computing system includes circuitry that can be used to implement a processing scheme for accelerating embedding layer processing of batches of inputs. The circuitry and processing scheme can be used to reduce or alleviate load imbalance on a multi-core processing unit, such as a distributed system of processors, by filtering out duplicate addresses in an address list used to retrieve inputs to an embedding layer of a neural network.

[0018] Having multiple duplicate addresses can be particularly problematic when computing or updating embeddings for a neural network. For example, samples in a batch input may include a variable number of features that need to be embedded, such as in the feature space of an embedding layer of a neural network. Each feature (or input) value has an embedding, such as a vector representing the values ​​of the neural network's trainable weights stored in the neural network's embedding table. Although each value in a sample (or batch) has an embedding, different samples in a batch (and even different features within the same sample) can include the same feature value. In some cases, even the same feature within a sample can include the same feature value.

[0019] For example, when computing or updating embeddings, it may be necessary to extract the same feature value for the same embedding table but for different features in a sample or batch, or for various features in different sample or batch inputs. Because the same input value may need to be extracted multiple times (e.g., from the same address in memory), a method for filtering out duplicate addresses when computing or updating embeddings can reduce the computational load on a distributed system and help improve or speed up the computation of the embedding layer of a neural network.

[0020] Deduplication techniques can be performed at the source or requesting processor before the addresses in the list are broadcast to different remote processors in a distributed system that are assigned to retrieve specific data elements for a given batch of inputs for processing in the embedding layer. The techniques described herein can be used to more efficiently compute the embedding output of a neural network layer from the retrieved data elements for the batch of inputs. For example, the process of generating the embedding output can include retrieving embeddings for one or more inputs in the batch and performing a reduction on the embeddings of relevant input features in the batch.

[0021] Relative to existing systems, the techniques described herein can be used to reduce or minimize the amount of computation performed to retrieve embedding data, perform the reduction, and return a reduced vector of embedding values. When generating embedding layer output, the techniques provide an improved method for balancing the computational load between a requesting processor, which processes reduced embedding values ​​for a batch of inputs, and a remote processor, which retrieves embeddings corresponding to the batch of inputs.

[0022] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Potential other features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims. Brief introduction of the attached figure

[0023] Figure 1 is a block diagram of an example computing system.

[0024] Figure 2 is a block diagram of example circuitry including an example multi-core processing unit.

[0025] Figure 3 An example host broadcasting a set of addresses to processors of an example multi-core processing unit is shown.

[0026] Figures 4A-4D Diagrams associated with an example processing scheme involving two or more processors of an example multi-core processing unit are shown.

[0027] Figure 5 An exemplary processor of an exemplary multi-core processing unit is shown.

[0028] Figure 6 is a flow chart illustrating an exemplary process for generating the output of a neural network layer.

[0029] Like reference numerals and numbers in the various drawings represent like elements. DETAILED DESCRIPTION

[0030] This specification describes techniques for accelerating embedding layer computations to mitigate processing delays that may occur due to processor load imbalance in a distributed computing system. For example, delays can occur during processor computations that generate the output of an embedding layer of a multi-layer neural network. Specifically, when a particular processor or core of a distributed system is required to perform a large number of lookup and computation operations (e.g., reducing or concatenating retrieved values) to generate an embedding, that particular processor may experience processing delays corresponding to the load imbalance.

[0031] The imbalance can be between a source processor that submits a query or a request to retrieve data for an input value and a remote processor that performs a lookup to retrieve the data. The data can be an embedding (e.g., a weight) of an embedded table, and the remote processor can be tasked to process the request to return the embedded data or weight value. For example, in a forward pass computation operation of the embedding layer, the remote processor processes a request from the source processor that causes the remote processor to look up an address to retrieve data stored in memory and perform a computation using the retrieved data. In existing distributed architectures, the remote processor performs minimal deduplication of the addresses specified in the request.

[0032] For example, in existing systems, a remote processor receives one or more duplicate addresses and performs data retrieval and related simplification operations corresponding to the input of all received addresses (including duplicate addresses). However, the architecture and processing schemes used by some existing systems are prone to load imbalance problems, especially when the computational load at one or more remote processors is unbalanced with the computational load at the source processor, or when there is an imbalance between two or more remote processors. These load imbalances can cause the computational operations that generate the output of the embedding layer to stop or significantly delay.

[0033] Typically, the embedding layer of a neural network is used to embed features into a feature / embedding space corresponding to the embedding layer. A feature can be an attribute or property shared by independent units on which analysis or prediction is to be performed. For example, an independent unit can be a set of words in a vocabulary or image pixels that make up an item such as an image and other documents. The algorithm for training the embedding layer embedding can be executed by the neural network processor to map the features to the embedding vector. In some embodiments, the embedding of the embedding table is learned together with other layers of the neural network that use the embedding. This type of learning occurs by backpropagating gradients to update the embedding table.

[0034] In other embodiments, embeddings can be learned independently of other layers of the neural network that use the embeddings, such as when the embeddings are pre-trained. For example, by processing information about discrete input features, a neural network processor can use the algorithm to calculate embeddings to determine a mapping or placement of similar inputs to embedding vectors that are geometrically close in the embedding space. In some cases, the process of calculating embeddings can represent a feature learning or feature engineering technique that allows the system to automatically discover the representations required for feature detection from raw input data.

[0035] In some embodiments, a given "input" can have one or more features of one or more types, and the embedding layer generates a corresponding embedding for each of these types. For example, the input can be for a search query with several different feature types. Feature types can include attributes of the user or user device (e.g., location, preferences, device type, etc.), query tokens, previously submitted queries, or other related types that can correspond to attributes of the search query. For any feature type with more than one feature for a given input, the computing system can be operated to retrieve a separate embedding for each of these features. The system can also be operated to combine the retrieved embeddings, for example by calculating the average of the embedding values, to generate a final embedding for that feature type.

[0036] Based on the context discussed above, this specification describes techniques for implementing a dedicated hardware circuit that is configured to perform address deduplication at a source processor before sending a list of addresses to a remote processor for lookup in a portion of memory allocated to the remote processor. The hardware circuit includes an identifier (ID) scatter circuit ("scatter circuit") that is configured to parse the original address list to filter out one or more "duplicate" addresses. The scatter circuit is configured to filter the duplicate addresses before bucketing the addresses belonging to each remote processor.

[0037] For example, a system including hardware circuitry may generate instructions to cause the decentralized circuitry to bucket addresses of data shards belonging to specific remote processors assigned to the system. The system then sends the address to the specific remote processor assigned to the data shard, which stores an embedding vector corresponding to the input (feature) stored at the address. The embedding vector may be a weight value or embedded portion of an embedding table stored on the data shard. In some cases, the embedding vector of the embedding table may be divided among multiple remote processors (processor nodes or cores) of the computing system. The operating system host is configured to send a set of addresses or vector indices to a subset of the remote processors to perform operations associated with retrieving a weight vector such as an embedding vector, which is mapped to a specific feature stored at the address of the set. In some examples, the remote processor performs one or more query and computation operations before providing or returning the weight vector to the system host.

[0038] The hardware circuitry includes a reverse mapper configured to augment or modify addresses by adding metadata, such as a hash key, to the addresses. The metadata is used to reverse map a specific set of retrieved or computed data (e.g., a collected vector) back to the corresponding original address before filtering duplicate addresses. In some cases, the circuitry is configured to perform a sparse reduction operation on data received from a remote core to form a dense matrix.

[0039] For example, before deduplication, the circuit is enhanced to iterate over each address in the original list of address locations and use the metadata tags generated by the reverse mapper to collect the vector outputs corresponding to the particular input address and perform a reduction operation on the collected vectors. These techniques allow a particular host processor processing multiple batches of inputs to perform computations in a distributed system of processors, rather than having them performed by remote processors mapped to a particular portion of memory that holds weight values ​​(e.g., parameters) corresponding to one or more inputs in the batch.

[0040] Figure 1 A block diagram of an example computing system 100 is shown, which is configured to retrieve data elements stored in a memory of the system 100 for performing embedding layer operations to generate an embedding set for training a neural network. In some cases, training the neural network can include determining one or more learning objectives based on an output vector, where the output vector represents a neural network embedding. For example, the embedding output can correspond to one or more output feature sets, where the output feature set includes a floating point / parameter vector of a given output dimension (64-d, 256-d, etc.).

[0041] When a neural network of the training system 100 performs certain computational functions, such as image or speech recognition, an embedding output is generated. In some embodiments, training the neural network includes updating a set of embeddings previously stored in an embedding table of the neural network, such as during a prior phase of training the neural network. In other words, the embeddings of the neural network can be pre-trained. For example, the embeddings of an embedding layer of a neural network may not be trained along with the neural network that is to use the embeddings. Therefore, the techniques described in this specification can be used to update the embeddings of a pre-trained neural network with improved efficiency compared to existing methods, such as methods used to pre-train embeddings.

[0042] The computing system 100 includes a host 102, a multi-core processing unit 104, and data slices 106a-106k, where k is an integer greater than 1. Generally, the host 102 can be a processing unit, such as a processor, multiple processors, or multiple processor cores. Thus, the host 102 can include one or more processors, and the host 102 can be operated to generate or process instructions to access a target dense matrix and send instructions 110 to the multi-core processing unit 104 to generate the target dense matrix. As described in more detail below, performing an embedding layer operation can include transforming sparse elements from one or more matrices to generate a dense matrix.

[0043] The multi-core processing unit 104 accesses corresponding elements 108a-108n from one or more of the data slices 106a-106k, where n is an integer greater than 1. The multi-core processing unit 104 generates a target dense matrix 112 using the corresponding elements 108a-108n and provides the target dense matrix 112 to the host 102 for further processing. The elements 108a-108n can be two-dimensional matrices of different sizes, and the multi-core processing unit 104 can generate the target dense matrix 112 by transforming each of the elements 108a-108n into a vector and concatenating the n vectors into a single vector.

[0044] Typically, in the case of embedding, the "sparse" information corresponding to the sparse elements can be a one-hot vector identifying the feature value. For example, if there are five possible values ​​for a given feature (e.g., A, B, C, D, E), the sparse vector identifies the feature value "A" as (1, 0, 0, 0, 0, 0), and the embedding layer maps (1, 0, 0, 0, 0) to a dense embedding vector for the feature value "A". In some embodiments, during training the embedding layer to learn the embedding, the elements 108a-108n can be weight values ​​of an embedding table that are converted into vectors, such as embedding vectors for feature values ​​"B" or "C". The weight values ​​can be transformed using a neural network processor of the multi-core processing unit 104, which executes a training algorithm to calculate the embedding based on at least the mapping of features to embedding vectors.

[0045] The host 102 can process an instruction to update a target dense matrix and send the updated dense matrix to the multi-core processing unit 104. For example, the target dense matrix can correspond to an embedding of a pre-trained neural network. Therefore, the host 102 can process an instruction to update the embedding of the pre-trained neural network, thereby generating an updated dense matrix. For example, during a subsequent iteration of training the neural network to update the embedding, a backward pass can be performed to update the embedding by determining a new mapping of input features to embedding vectors and generating an updated dense matrix based on the new mapping. In some embodiments, the multi-core processing unit 104 can be operated to convert the updated dense matrix into corresponding sparse elements and update one or more sparse elements (e.g., weights) stored in the data slices 106a-106k accordingly.

[0046] As described above, the host 102 is configured to process instructions executed within the computing system 100. In some embodiments, the host 102 is configured to process the target dense matrix 112 generated by the multi-core processing unit 104. In some other embodiments, the host 102 may be configured to request the multi-core processing unit 104 to generate the target dense matrix 112, and another processing unit may be configured to process the target dense matrix 112.

[0047] Each processor of the multi-core processing unit 104 is configured to retrieve data elements stored in the memory of the system 100. The memory may include multiple data slices 106a-106k that store data including elements 108a-108n. The data may include inputs, activations, gain values, or weight values ​​corresponding to parameters of a weight matrix structure or kernel. In some embodiments, the data slices 106a-106k may be a volatile memory unit or multiple units. In some embodiments, the data slices 106a-106k may be a non-volatile memory unit or multiple units. The data slices 106a-106k may also be another form of computer-readable medium, such as a device in a storage area network or other configuration. The data slices 106a-106k may be coupled to the multi-core processing unit 104 using an electrical connection, an optical connection, or a wireless connection. In some implementations, the data shards 106a-106k may be part of a multi-core processing unit 104 and based on a processor-in-memory (PIM) architecture.

[0048] The multi-core processing unit 104 is configured to determine a dense matrix based on the sparse elements. In some embodiments, the multi-core processing unit 104 can be configured to determine the location of the sparse elements based on the dense matrix. The multi-core processing unit 104 includes a plurality of interconnected processors or processor cores. For example, the multi-core processing unit 104 can be a distributed processing system including a plurality of interconnected processor cores. In general, the terms "processor" and "processor core" are used interchangeably to describe the discrete, interconnected processing resources of the multi-core processing unit 104.

[0049] Figure 2 is a block diagram of an exemplary dedicated circuit 200 including an exemplary multi-core processing unit 104. In general, computing system 100 may be implemented using one or more circuits 200 (described below). Figure 2 In one embodiment, the multi-core processing unit 104 includes a set of interconnected processors including processors 0 through 15. As described in more detail below, each of the processors 0 through 15 of the multi-core processing unit 104 can be assigned to one or more of the data shards 106a through 106k to obtain data (e.g., feature values ​​and weights) stored in memory locations of the data shards assigned to the processors. Each processor of the multi-core processing unit 104 can be operated to fetch or retrieve data from a memory location identified in a request received from the host 102 or from another processor in the multi-core processing unit 104.

[0050] Processors collaborate to retrieve data elements mapped to the embedding table. The data elements are retrieved to perform embedding layer operations, such as calculations for generating an output vector (e.g., a dense vector) representing an embedding of the neural network. In some embodiments, the embedding output is a mapping of discrete inputs in a batch of inputs to a dense vector of numerical values ​​(e.g., floating point values), which can be used to define relationships between objects such as terms or words in a vocabulary. For one input in a given batch, the embedding can be a vector, while for the entire batch of inputs, the embedding can be a matrix, such as a collection of vectors. In some embodiments, the vector of embedding values ​​corresponds to trainable weights learned by the neural network during training. The learned weights can encode one or more words by looking up a dense vector stored in the embedding table, where the dense vector corresponds to a specific word. Multiple dense vectors can correspond to a dense matrix that is used to train the neural network to perform a specific task or function related to recognizing a sequence of words, such as language processing.

[0051] In some examples, embedding layer calculations are large-scale data lookup problems typically used during the training phase of a neural network. For example, during training, a large table of weight values ​​may be sharded or partitioned across multiple dimensions of a multidimensional tensor, such as the x-, y-, or z-dimensions of the multidimensional tensor. Specifically, the table of weight values ​​may be sharded across data shards 106a-106k such that each weight value is stored at a corresponding location in a memory comprising the data shards.

[0052] Furthermore, one or more of the data shards 106a-106k can be partitioned to specific processors 0-15 in the multi-core processing unit 104. Each shard or partition of the table corresponds to a data shard 106k assigned to a specific processor or processor core of the multi-core processing unit 104. In some examples, each processor core is a respective neural network processor. In other examples, each processor core is a respective core of a single neural network processor. In general, the computing system 100 may include multiple processors or multiple processor cores based on the desired configuration of the system.

[0053] Reference again Figure 2 and hardware circuit 200, the circuit includes an identifier (ID) dispersal circuit 202 ("dispersal circuit 202"), which includes a hash generator 204 and a reverse mapper 206. Each of hash generator 204 and reverse mapper 206 is described below.

[0054] The scatter circuit 202 is configured to process a raw address list that may be received by the host 102. The raw address list identifies a memory location in a memory of the system 100, such as a memory including data slices 106a-106k. Figure 2 , the dispersion circuit 202 is shown as being external to or separate from the host 102 and the multi-core processing unit 104. However, in some embodiments, the dispersion circuit 202 may be a part of the host 102, such as a dispersion unit or computing module, or a sub-circuit of the host 102.

[0055] Similarly, in some embodiments, the scatter circuit 202 can be included in the multi-core processing unit 104 as part of a processor in the multi-core processing unit 104. For example, the scatter circuit 202 can be included in a source / master processor (e.g., processor 0) of the multi-core processing unit 104. The source processor can be tasked by the host 102 to generate one or more requests and provide the requests to multiple other processors (e.g., processors 1-15) in the multi-core processing unit 104 to retrieve data elements from memory locations of data slices 106a-106k assigned to these processors.

[0056] The dispersal circuit 202 communicates with the host 102 to receive the raw address list. For example, the host 102 receives a batch of inputs to a neural network layer, such as from an external source or a controller of the system 100 that manages machine learning operations performed by the system. For example, the machine learning operations can be used to train a neural network to process image or audio data, such that the neural network can learn how to recognize specific objects in an image / video or how to detect certain word sequences in an audio stream.

[0057] Host 102 receives one or more batches of input. In some embodiments, host 102 receives multiple batches of input corresponding to one or more samples. For example, a first set of multiple batches of input may correspond to one sample 210-1, while a second set of multiple batches of input may correspond to another sample 210-2. Each sample 210-1, 210-2 may include an input corresponding to a set of features of an object (e.g., an image or an audio stream). For example, each of samples 210-1 and 210-2 may include a respective batch of input features corresponding to features of different regions of the same image, or corresponding to different regions of multiple different images.

[0058] Each input in a batch of inputs is stored in a memory location, and the location can be identified by an address in the original list of addresses received by the scatter circuit 202 from the host 102 (e.g.,

[99] ,

[736] ,

[37] ,

[99] , etc.). In some embodiments, each of the inputs has a corresponding set of weights, which can be stored as data or sparse elements of an embedding table that is sharded across data shards 106a-106k and partitioned across processors 0 through 15. In some embodiments, the embeddings for the various features in the batch of inputs are stored in a distributed manner, e.g., across the various data shards of the system 100.

[0059] For example, the features can be distributed across multiple processor nodes or cores (0-15), where each node is allocated a memory location for storing the locations of the embedded slices. As described in more detail below, the host 102 (e.g., a source processor) provides a request including one or more address subsets to one or more of the processors 0-15 (e.g., remote processors) for processing to retrieve data elements for generating a dense matrix.

[0060] As described above, in some existing systems, a remote processor that receives a duplicate address performs data retrieval, arithmetic operations (e.g., multiplication), and associated reduction operations on the received address that includes the duplicate address. This can lead to load imbalance problems, particularly when the computational load at the remote processor is unbalanced with the computational load at the requesting or source processor.

[0061] The scatter circuitry is configured to parse the original address list to filter out one or more "duplicate" addresses. The address list received by the host 102 includes one or more duplicate addresses corresponding to the input characteristics in the batch. For example, the original address list received at the system 100 may include the addresses

[26] ,

[96] ,

[99] , [1], [7],

[312] ,

[99] , and

[912] of the batch input. In this example, the original address list includes address

[99] as a duplicate address in the list. The scatter circuitry 202 is configured to filter out duplicate addresses before bucketing the addresses belonging to each remote processor and providing a request to the remote processor to retrieve the data element corresponding to the grouped address.

[0062] Scattering circuit 202 identifies one or more duplicate addresses in the address list for each input in the batch of inputs. For example, a single embedding table may be referenced by multiple features. Thus, duplicate addresses may exist on multiple different features. For example, embedding table T may be referenced by features F1 and F2. In this example, an address (e.g.,

[99] ) may be duplicated within the input of F1 (in one or more samples) and within the input of F2 (in one or more samples).

[0063] Scatter circuitry 202 is configured to perform deduplication on the original addresses by filtering out duplicate addresses and providing unique addresses (identifiers) to multi-core processing unit 104 for processing at one or more of processors 0-15. For each duplicate address, scatter circuitry 202 generates an identifier that uniquely identifies the duplicate address in a list of original addresses. Scatter circuitry 202 then filters the list of original addresses to generate a filtered list of addresses based on certain identifiers that indicate duplicate addresses in the list of original addresses, such that the filtered list of addresses includes only unique identifiers corresponding to non-duplicate addresses.

[0064] More specifically, for each address in the original list, the scatter circuit 202 uses the hash generator 204 to generate an identifier corresponding to a hash ID. The memory of the system 100 may have address values ​​(e.g.,

[99] ) with different memory offsets (e.g., offset 0, offset 1, etc.). The hash generator 204 may generate the hash ID based on the address value, the memory offset of the address value, or a combination of each. For example, for repeated occurrences of an address in the list, a portion of the hash ID generated based on the address value (e.g.,

[99] ) will be repeated, but the portion of the hash ID will be different from the second portion of the hash ID generated based on the memory offset, or based on a combination of the address value and the memory offset. In some examples, the scatter circuit 202 may append the corresponding first portion of the hash ID to the address in the original list as a prefix of the address. The scatter circuit 202 may recursively scan the address list to identify repeated occurrences of the appended prefix and filter out the particular address based on the repeated occurrences.

[0065] In some embodiments, the additional prefix, or the first portion of the hashed ID, is an example of a metadata tag that is used by the scatter circuitry 202 to enhance the addresses in the original list to generate a filtered list of addresses. Based on the unique identifier generated by the hash generator 204, the scatter circuitry 202 generates a filtered list of addresses for the batch input. The filtered list of addresses includes only unique identifiers or addresses (e.g., non-duplicate addresses) of the memory locations of the data shards 106. The filtered list of addresses is then provided to the multi-core processing unit 104 so that the processors 0-15 receive only unique addresses and not multiple occurrences of duplicate addresses.

[0066] The scatter circuit 202 uses a reverse mapper 206 to generate a reverse map, which is used to map the collected vectors back to specific addresses in the original list. The reverse mapper 206 is configured to add one or more metadata tags to the address or batch input to enable reverse mapping. For example, the reverse mapper 206 can enhance or modify the addresses in the original list by adding metadata to the addresses. The metadata can be a metadata tag including a hash ID or an annotation specifying the memory offset of the address. Before filtering duplicate addresses, the reverse mapper 206 uses the metadata to reverse map a specific set of retrieved or calculated data (e.g., collected vectors) back to the original address.

[0067] In this way, the circuit 200 provides dedicated hardware that is operable to traverse each address in the original list of address locations before deduplication and collect the vector output corresponding to the particular input address using the metadata tag generated by the reverse mapper 206. As described in more detail below, the circuit 200 is operable to perform a reduction operation on the collected vectors to generate a dense matrix. For example, an example source processor of the host 102 or multi-core processing unit 104 may include an exemplary sparse reduce circuit that is configured to perform a reduction of data elements received from a remote core to form a dense matrix. Reference will be made below to Figure 5 Describe this in more detail.

[0068] Figure 3 1 shows a host 102 broadcasting a set of addresses to each of processors 0, 4, 8, and 12 of a multi-core processing unit 104. Figure 3 As shown, host 102 sends a unique identifier or address of a batch input to a processor of multi-core processing unit 104. In some embodiments, host 102 may send a unique identifier for each batch input corresponding to sample 210-1 and each of sample 210-1. For example, host 102 may send the address [0, 26, 96, 99] of batch input 302 to processor 12 and the address [2, 9, 17, 736] of batch input 304 to processor 0. Figure 2 As shown, batch input 302 is addressed for sample 210 - 1 , while batch input 304 is addressed for sample 210 - 2 .

[0069] In some embodiments, host 102 may be operated to provide different batches of input to different processors and different ranks of processors in multi-core processing unit 104. For example, host 102 may send different batches of input to different processors in a sequential manner or in parallel. Similarly, host 102 may send different batches of input to different ranks of processors in a sequential manner or in parallel.

[0070] Figures 4A-4D Each of shows a respective diagram associated with an example of a processing scheme 402 that may be implemented at system 100 using at least host 102 and two or more processors in multi-core processing unit 104 .

[0071] As described above, processors 0-15 collaborate to retrieve data elements mapped to an embedding table, such as a sparse embedding table including a plurality of sparse elements. The data elements are retrieved in response to a remote processor processing a request provided by a source (or requesting) processor. In some cases, the data elements are retrieved to perform an embedding layer operation, such as a calculation to generate an output vector (e.g., a dense vector) representing a neural network embedding, or a calculation to generate partial activations in response to multiplying an input (e.g., a gain value or activation value) by a weight vector to generate partial activations.

[0072] As described above, the embedded table can be a table of weight values ​​that is sharded between the data shards 106a-106k such that each weight value is stored in a corresponding location in memory. The data shards 106a-106k can be partitioned to specific processors 0-15 in the multi-core processing unit 104 such that the weight values ​​of the embedded table correspond to the data shards assigned to the specific processors 0-15 of the multi-core processing unit 104.

[0073] As described below (and reference Figure 5 ), the request provided by the source processor to the remote processor to retrieve a data element may include an address (e.g.,

[912] ) identifying a memory location of a data slice. The memory location stores data for an input, such as a gain value or activation. The request is provided to cause the remote processor to retrieve a data element representing an input, such as one of a plurality of inputs in a batch of inputs received at the host 102. In some embodiments, the request is provided to cause the remote processor to perform a lookup operation to retrieve a specific or unique weight corresponding to a specific input in the batch of inputs.

[0074] exist Figure 4A In the example of FIG, block 402 illustrates an exemplary processing scheme in which at least processors 0-3 collaborate to retrieve a data element mapped to a portion of an embedded table. In this example, processor 0 may be a source (or requesting) processor that submits a corresponding request to each of processors 1, 2, and 3. More specifically, processor 0 is a source processor, and each of processors 1, 2, and 3 is a remote processor relative to processor 0.

[0075] In some cases, processors 0, 1, 2, and 3 form a subset 404. In this subset 404, each of processor 0, processor 1, processor 2, and processor 3 can be a source processor relative to the other processors in subset 404, where the other processors are remote processors that receive and process requests from the source processors in subset 404. In some cases, the source processors and remote processors in subset 404 can still receive and process requests from another processor in multi-core processing unit 104 or from host 102, where host 102 can be a primary, source processor that is not included in the subset.

[0076] The exemplary scheme illustrated in block 402 can be part of a processing technique implemented at system 100 to generate an output of a neural network layer, such as an embedding output of an embedding neural network layer or an output of a first hidden layer of a neural network. The processing technique includes one or more remote processors that provide data elements comprising a plurality of inputs and corresponding query values ​​to a requesting processor. The data elements are stored in memory locations of data shards assigned to the remote processors.

[0077] In some embodiments, in response to each remote processor performing a lookup, a data element is provided to the source processor to obtain a weight vector, the weight vector including a corresponding weight value to be multiplied by a particular input of the plurality of inputs. For example, a portion of memory can be allocated to the remote processor that stores weights of a kernel matrix to be multiplied by activations at corresponding addresses stored in a list of addresses received by the source processor. The inputs are multiplied by the weight values, for example, in a subsequent neural network layer (see at least the following Figure 4D Example operations will be described to further illustrate the processing technique.

[0078] During exemplary operation of system 100 processing multiple batches of input, an original list of multiple addresses is received by host 102 or another source processor, such as processor 0 of multi-core processing unit 104. Address

[912] may be included among multiple addresses in the list. For a given batch or sample,

[912] may appear more than once in the received list. Specifically, address

[912] may be one of multiple addresses that are replicated between (i) a set of input features or (ii) a sample of input features. In some embodiments, each repeated occurrence of address

[912] in a batch or sample may be used for a specific memory offset in a given data shard.

[0079] Address

[912] can be a location in memory where an input / feature value is stored, where the input / feature value is an identifier (e.g., a hot identifier) ​​of a word in a given vocabulary, such as the word "car". For example, an embedding table can map a hot identifier of a word to an embedding. More specifically, address

[912] can store instances of input features that appear multiple times within the feature values ​​of a given sample and / or within a given batch of inputs. Each occurrence of address

[912] can be a feature with an embedding, such as a particular weight or vector of trainable weight values ​​stored in an embedding table in memory.

[0080] The system 100 can perform deduplication on the original list to remove multiple occurrences of the address

[912] to generate a filtered address list that includes only unique addresses. The reverse mapper 206 can attach a metadata tag to a particular occurrence of the address

[912] in a batch or sample to map the collected vector (e.g., weight vector) back to the particular occurrence of the address

[912] in the original list. As described in more detail below, an example source processor can use this reverse mapping to locally perform a computation, such as multiplying between (i) an input value stored at that particular occurrence of the address

[912] and (ii) a collected vector received from a remote processor.

[0081] In general, the address

[912] can identify a memory location of a data shard assigned to processor 0 that stores a particular input value. In some implementations, the address

[912] in the data shard assigned to processor 0 can also be duplicated, e.g., 100 times in each of multiple batches of input, but to refer to different data shards assigned to other processors of the processing unit 104. For example, the address

[912] can be a particular address that is common in respective data shards assigned to one or more processors in the multi-core processing unit 104.

[0082] An example source processor, such as the host 102 or processor 0, can perform 16 memory lookups, one for each processor in the multi-core processing unit 104 of the system 100, but would not perform the computations related to reduction or matrix multiplication. In Figure 4A In the example of 2C (and 4C discussed below), processor 3 can send only its respective input value (e.g., activation or gain) and weight value to processor 0, which correspond to a unique address that can be generated for multiple duplicate occurrences of the address

[912] in a particular batch.

[0083] In the example of 2C (and 4C discussed below), processor 3 can send only its respective input value (e.g., activation or gain) and weight value to processor 0, which correspond to a unique address that can be generated for multiple duplicate occurrences of the address

[912] in a particular batch. Figure 4B In the example of 2C (and 4C discussed below), processor 3 can send only its respective input value (e.g., activation or gain) and weight value to processor 0, which correspond to a unique address that can be generated for multiple duplicate occurrences of the address

[912] in a particular batch.

[0084] For example, processor 3 may be a respective remote processor in a subset of processors including processor 0, processor 1, and processor 2. In this example, processor 0 may be a requesting processor in a subset including processor 3, which is a remote processor relative to at least processor 0. In this manner, processor 3 is configured to look up and locally reduce a data element, such as a unique weight value for an input stored at address

[912] , to generate an exemplary weight vector [f2].

[0085] exist Figure 4C In the example of FIG. 1 , a source or requesting processor (e.g., processor 0) receives locally reduced data elements at each remote processor and performs one or more operations on these data values. In some embodiments, the requesting processor is configured to perform additional reduction operations on the received values. For example, when processor 3 retrieves and locally reduces the input stored at address

[912] and the corresponding lookup data to generate the value of vector [f2], the system 100 can be operated to determine the reduction of the input relative to the corresponding sample or batch of inputs to which the input is assigned.

[0086] For example, using request processor 0 of subset 404, system 100 may determine that input

[912] needs to be reduced by a factor of five for sample 210-1 and by a factor of two for sample 210-2. In addition, request processor 0 is configured to perform a reduction on all data values ​​received from each remote processor in subset 404. For example, request processor 0 is configured to locally reduce each of weight vector [f2], weight vector [g1], and weight vector [f1] received from remote processor 1, remote processor 2, and remote processor 3, respectively. In some embodiments, request processor 0 is configured to look up its own data elements, such as the unique weight values ​​of the inputs stored at addresses [1], [7], [2], [9], and

[17] , and locally reduce the lookup values ​​on these features to generate example weight vectors [f0] and [g0]. The request processor may perform the reduction on both its own lookup values ​​and the data values ​​received from each remote processor in subset 404.

[0087] exist Figure 4CIn the example of , each of subsets 406 and subsets 408 can be used to implement a processing scheme similar to or the same as the processing scheme 402 described above. For example, in subset 404, processor 7 is configured to find and locally reduce a remote processor of data elements, such as unique weight values ​​of an input stored at address

[37] , to generate an exemplary weight vector [e1]. In subset 406, the source or request processor (e.g., processor 4) receives the data elements of the locally reduced vector [e1] at the remote processor 7 and performs one or more operations on these data values. Processor 4 can then generate exemplary weight vectors [d0] and [e0] based on one or more concatenation or reduction operations performed on the embeddings retrieved for various input values ​​in the batch or sample.

[0088] In some embodiments, each of processors 1-15 may send its respective input value (e.g., activation or gain) and weight value to processor 0, the input value and weight value corresponding to a unique address that may be generated for multiple repetitions of address

[912] in a particular batch. In other embodiments, each of processors 0-15 may be assigned a respective batch for processing, and each processor will perform its respective calculation for 100 repetitions of address

[912] in its respective batch.

[0089] Figures 4A-4D The processing techniques described herein allow a particular host processor processing multiple batches of inputs to perform computations in a distributed system of processors, rather than having computations performed by remote processors mapped to a specific portion of memory that holds weight values ​​(e.g., parameters) corresponding to one or more inputs in the batch. These techniques may also translate into improved load balancing and scalability of system 100.

[0090] exist Figure 4D In the example of , processor 12 can represent a processing node of an example distributed processing system of system 100. In some embodiments, processor 12 can be a master processor for processing multiple batches of inputs. Processor 12 is configured to receive each of the vector outputs (e.g., dense vectors) that are reduced by a corresponding request processor 104n in the multi-core processing unit 104. More specifically, processor 12 can receive a vector output corresponding to a data element of each output matrix (representing an embedding of a neural network) that is locally reduced by a corresponding remote processor.

[0091] The processor 12 is configured to perform a calculation to produce partial activations in response to multiplying an input, such as a gain value or an activation value, by a weight value of a weight vector, thereby obtaining partial activations. For example, the processor 12 is configured to generate one or more sets of partial activations based on multiplications between the input and the weight values ​​of the weight vectors [a] and [b].

[0092] Figure 5 An example processor 104n of the multi-core processing unit 104 is shown, where n is an integer greater than 1. For example, the processor 104n may be the processor 104n described above with reference to FIG. Figure 2 Generally, processor 104n is configured to receive a request 542 from a network of nodes 520 to fetch or retrieve data (eg, sparse elements) stored in one or more data shards 530.

[0093] For example, for a dense matrix generated using sparse elements, the source processor 502 sends a request 542 to the processor 104n in the node network 520. The node network 520 can be a two-dimensional mesh network. In some embodiments, the request 542 to extract data includes an instruction to transform the extracted sparse elements into a dense matrix. As described above, the request to transform the extracted sparse elements into a dense matrix can be performed during training of the embedding layer to learn the embedding. For example, the extracted elements can be weight values ​​of an embedding table, which are transformed into an embedding vector based on at least a mapping of input features to embedding vectors. The source processor 502 can broadcast the request 542 to the processor 104n. The routing of the broadcast request 542 can be similar to the routing of the host 102 that provides the request to one or more processors 104n. Figure 2 In some embodiments, the source processor 502 may be the host 102 or another processor 104 n included in the multi-core processing unit 104 .

[0094] In general, the processor 104n is configured to receive a request 542 to retrieve a data element stored in one or more data shards 530 and to determine whether the processor 104n is assigned to access an address of a location in memory that stores the data indicated by the request 542. In some embodiments, the processor 104n may determine whether it is assigned to access the data element indicated by the request 542 by using a lookup table. For example, if the address of the particular requested data element (e.g., the address at memory offset 1 of lookup table No. 5

[99] ) is included in the lookup table, the processor 104n may send a signal to an example data retrieval unit of the processor to cause the data retrieval unit to retrieve the particular requested data element. In some embodiments, the processor 104n may be configured to broadcast the received request 542 to another processor 104n on the node network 520.

[0095] The processor 104n is configured to retrieve one or more requested data elements from a particular data slice 530 assigned to the processor 104n in response to a processing request 542. In some embodiments, the processor 104n is a vector processor, a vector processing unit (VPU), an array processing unit, or any suitable data computing resource for processing arrays and matrices of vector elements. In some embodiments, each of the processors 104n included in the multi-core processing unit 104 can be assigned to a particular data element stored in the data slice 530, such as a particular data element of an embedded table, based on an address and a corresponding memory location for storing the data element.

[0096] In some embodiments, the source processor 502 may generate one or more requests 542 for data elements corresponding to multiple batches of input to be retrieved and processed using one or more processors 104n. In some cases, each batch of input may be associated with a corresponding request 542 for a subset of the requested data elements. Each processor 104n is configured to independently retrieve the assigned data elements from its assigned data slice 530. In some embodiments, as described in more detail below, the processor 104n causes the retrieved data to be forwarded to one or more units of the processor for further processing.

[0097] like Figure 5 As shown, processor 104n includes a sparse reduction unit 506, a concatenation unit 508, and a compression / decompression unit 510. Processor 104n can forward the retrieved data elements to sparse reduction unit 506. Sparse reduction unit 506 is configured to reduce the dimensionality of the retrieved data elements. For example, processor 104n can generate vector elements with dimensions of 100×1. Sparse reduction unit 506 is operable to receive the extracted data elements with dimensions of 100×k and reduce the dimensionality of the extracted data elements 346 to 100×1 through logical operations, arithmetic operations, or a combination of both to generate sparse reduced elements 548. Sparse reduction unit 506 is configured to output sparse reduced elements 548 to concatenation unit 508.

[0098] The concatenation unit 508 is configured to rearrange and concatenate the sparse reduction elements 548 to generate the concatenation element 550. For example, the processor 104n can be configured to access a data element comprising an input to a neural network layer and a corresponding set of weight values ​​(e.g., sparse elements No. 1 to No. 100 of database table No. 1). Before returning the retrieved sparse element No. 5 (e.g., the second weight value), the processor 104n can be operated to return the retrieved sparse element No. 10 (e.g., the first weight value) to the sparse reduction unit 506. The concatenation unit 508 is configured to rearrange the later received sparse element No. 5 to be sorted before the earlier received sparse element No. 10, and is configured to concatenate the sparse elements No. 1 to No. 100 as the concatenation element 550.

[0099] The compression / decompression unit 510 is configured to compress the concatenated elements 550 to generate a dense matrix 552 of the node network 520. For example, the compression / decompression unit 510 can be operated to compress the zero values ​​in the concatenated elements 550 to improve the bandwidth of the node network 520. In some cases, multiple zero values ​​can be generated in response to an arithmetic operation between the input and the weight, such as a multiplication between the input and the weight value having zero values, which produces a matrix of concatenated elements 550 having multiple zeros as a result of the multiplication. The compression / decompression unit 510 can compress the matrix to generate the exemplary dense matrix 552 of the node network 520.

[0100] In some embodiments, the compression / decompression unit 510 may decompress the received dense matrix. For example, the first processor 104n (e.g., Figure 2 The processor 9 in the example can receive data from an adjacent second processor 104n (e.g., Figure 2 The adjacent second processor 104n may decompress the received dense matrix and concatenate the decompressed dense matrix with the concatenation element 550 to form an updated concatenation element that may be compressed and then output to the node network 520.

[0101] Figure 6 6 is a flow chart illustrating an exemplary process 600 for generating an output of a neural network layer. Process 600 can be implemented or performed using the system 100 described above. The description of process 600 can refer to the computing resources of the system 100 described above. In some embodiments, the steps or actions of process 600 are enabled by programmed firmware or software instructions that can be executed by one or more processors and resources of the devices described in this document.

[0102] Referring now to process 600, a host computer of system 100 receives a batch of inputs for a neural network layer (602). For example, host computer 102 may receive one or more batches of inputs for processing by multi-core processing unit 104. Each input in one or more batches of inputs is stored in a memory location identified by an address. For at least one batch of inputs, circuitry of system 100 identifies one or more duplicate addresses in a list of addresses for each input in the batch of inputs (604). For example, dispersal circuitry 202 may be operable to scan the list of addresses to identify one or more duplicate addresses in the list of addresses for each input in the batch of inputs.

[0103] For each duplicate address identified, the circuitry generates a unique identifier that identifies the duplicate address in the address list (606). For example, the dispersal circuitry 202 may be operated to generate a corresponding unique identifier for each duplicate address to identify a particular address as a duplicate in the address list. The address list may be used to form a batch input of a sample of input features, the input features comprising a plurality of groups of input features. The circuitry of the system 100 may be operated to generate a unique identifier for each duplicate address in a corresponding group of input features in the input feature sample.

[0104] Based on the one or more unique identifiers generated by the circuit, the system 100 generates a filtered list of addresses for the batch input (608). For example, the host 102 interacts with the dispersion circuit 202 to receive a corresponding unique identifier for each duplicate address and generates the filtered list of addresses for the batch input based on at least the one or more unique identifiers generated by the dispersion circuit 202.

[0105] For the batch of inputs, the system 100 obtains a first input from a memory location identified by an address corresponding to the unique identifier based on the filtered address list (610). For example, based on the filtered address list, the system 100 may use a specific processor assigned to the corresponding data shard to obtain the first input from a memory location of the data shard, the memory location identified by the address corresponding to the unique identifier.

[0106] To obtain the first input, the request processor may provide a request to the remote processor for processing by the remote processor to obtain the first input. The request may include an address of the first input corresponding to the unique identifier. In some cases, the request and address may be sent separately but simultaneously or in parallel. In some cases, the request and address may be sent sequentially, with one preceding the other.

[0107] In response to the remote processor processing the request, the request processor receives a first input from the remote processor, the first input stored at a memory location of a data shard assigned to the remote processor. In some embodiments, the request processor is operable to perform one or more reduction operations to generate an output of the neural network layer from the first input, as described in more detail below, the first input being obtained from the data shard.

[0108] System 100 may determine a partitioning of addresses in the filtered list between each of the plurality of processors in multi-core processing unit 104. For example, system 100 may determine that the data element associated with address

[26] is partitioned into a data slice assigned to processor 13, that the data elements associated with addresses

[96] and

[99] are partitioned into a data slice assigned to processor 15, or that the data element associated with address

[912] is partitioned into a data slice assigned to processor 3.

[0109] In this manner, and for each partition of addresses, the system can retrieve a first input from a memory location identified by an address in the address partition using a corresponding processor and other corresponding data elements (e.g., weight values) in the data slice, the corresponding processor being assigned to retrieve the input, and the data slice being assigned to the corresponding processor. For example, each partition of addresses can be assigned to a data slice having an activation value or gain value corresponding to a vector element of a multidimensional sparse tensor. In some embodiments, each data slice having an activation or gain value is assigned to a specific processor for multiplication with the corresponding weight vector to perform a neural network computation.

[0110] System 100 generates an output of the neural network layer from the obtained first input, such as an embedding output (612). In some embodiments, the neural network layer is an embedding layer of a neural network, and the embedding output includes an embedding feature vector. Dispersion circuit 202 is configured to determine a reverse mapping of unique identifiers to duplicate addresses for a particular input in a batch of inputs to the embedding layer. In some embodiments, the reverse mapping of unique identifiers to duplicate addresses is determined for a particular input across multiple sets of input features.

[0111] Based on the reverse mapping of the unique identifier to the duplicate addresses of the particular input, the system 100 may be operated to generate one or more vectors (e.g., embedding feature vectors) as outputs of the embedding layer. For example, generating the output of the neural network layer (e.g., embedding output) based on the reverse mapping may include analyzing a list of addresses that resolve duplicate addresses to map an embedding feature vector back to a particular input in the batch of inputs to the neural network layer, the embedding feature vector being generated for the duplicate address and corresponding to the unique identifier.

[0112] In some examples, generating the output of the embedding layer may include generating one or more dense matrices in response to processing data associated with a sparse table of embedded values. The data associated with the table may be represented in the form of a matrix. The processor and computing resources of the system 100 may be used to manipulate the data based on linear algebra algorithms including matrix multiplication. The matrix may be a one-dimensional vector or a multi-dimensional matrix. The matrix may be represented by a data structure, such as a database table or a variable. The embedding table may be sharded in, for example, row and column dimensions and stored in a portion of memory that is mapped to a specific processor among the multiple processors forming the multi-core processing unit 104.

[0113] The subject matter and functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by, or to control the operation of, data processing apparatus.

[0114] Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to appropriate receiver equipment for execution by data processing equipment. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0115] The term "computing system" includes various devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0116] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, or component, subroutine, or other unit suitable for use in a computing environment.

[0117] A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that stores other programs or data, for example, in a markup language document, in a single file dedicated to the program, or as one or more scripts in multiple coordinated files, such as files storing one or more modules, subroutines, or code portions. A computer program may be deployed for execution on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0118] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPGPU (general purpose graphics processing unit), and apparatus can also be implemented as special purpose logic circuitry.

[0119] For example, the computer that is suitable for executing computer program can be based on general or special purpose microprocessor or both or any other type of central processing unit.Usually, central processing unit will receive instruction and data from read-only memory or random access memory or both.Some elements of computer are central processing unit for carrying out and implementing instruction and one or more memory devices for storing instruction and data.Usually, computer will also include or be operably coupled to receive data from one or more large-capacity storage devices or transmit data to one or more large-capacity storage devices, and described large-capacity storage devices are used to store data, for example magnetic disk, magneto-optical disk or optical disk.However, computer need not have such equipment.In addition, just give a few examples, computer can be embedded in another device, for example mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver or portable storage device, for example universal serial bus (USB) flash drive.

[0120] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CDROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0121] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as an LCD (liquid crystal display) monitor, to display information to the user, and a keyboard and pointing device, such as a mouse or trackball, to allow the user to provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. Furthermore, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0122] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server, or includes middleware components, such as an application server, or includes front-end components, such as a client computer with a graphical user interface or a web browser, through which a user can interact with an implementation of the systems and techniques described herein, or any combination of these back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0123] Computing systems can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises by virtue of computer programs running on their respective computers and having a client-server relationship to each other.

[0124] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope of what is claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as working in certain combinations and even initially claimed as such, in some cases one or more features in the claimed combination may be removed from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0125] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0126] Specific embodiments of the present subject matter have been described. Other implementations are within the scope of the following claims. For example, the actions in the claims can be performed in a different order and still achieve the desired results. As an example, the processes illustrated in the accompanying drawings do not necessarily require the specific order or sequence shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method for performing neural network computations using a system configured to implement a neural network on a hardware circuit, characterized in that: The method comprises: A host receives a batch of inputs to a neural network layer, wherein each input in the batch of inputs is stored in a memory location identified by an address; identifying one or more duplicate addresses in a list of addresses for one or more inputs in the batch input; For each duplicate address: generating a unique identifier, the unique identifier identifying the duplicate address in the address list; For the batch of inputs, obtaining a first input from a memory location identified by an address corresponding to the unique identifier; An embedding output of the neural network layer is generated from the obtained first input.

2. The method according to claim 1, characterized in that The neural network layer is an embedding layer of the neural network, the embedding output comprises an embedding feature vector, and the method further comprises: The system's dispersal circuitry determines a reverse mapping of the unique identifier to a repetition address of a particular input in the batch of inputs to the embedding layer; and Based on the reverse mapping of the unique identifier to the repeated address of the specific input, one or more embedding vectors are generated as outputs of the embedding layer.

3. The method according to claim 2, characterized in that The method comprises: generating a filtered list of addresses for the batch input based on one or more unique identifiers; determining, for addresses in the filtered address list, an address partition between each of a plurality of processors; and Wherein obtaining the first input comprises: for each address partition, using a corresponding processor assigned to retrieve input from the address, obtaining the first input from a memory location identified by the address in the address partition.

4. The method according to claim 2, characterized in that Wherein generating the embedding output of the neural network layer based on the reverse mapping comprises: The address list including the duplicate address is parsed to map the embedded feature vector corresponding to the unique identifier generated for the duplicate address back to the specific input in the batch of inputs to the neural network layer.

5. The method according to claim 2, characterized in that The method comprises: The request processor provides the remote processor with an address of a first input corresponding to the unique identifier and a request for the first input; The request processor receives the first input from the remote processor, the first input being stored in a memory location of a data slice assigned to the remote processor; and One or more reduction operations are performed to generate the embedding output of the neural network layer from a first input obtained from the data slice.

6. The method according to claim 2, characterized in that The address list is used to form a batch input of input feature samples including multiple groups of input features, and generating the unique identifier includes: The unique identifier is generated for each repeated address on a corresponding set of input features in the input feature sample.

7. The method according to claim 6, characterized in that Determining the reverse mapping of the unique identifier to the duplicate address includes: The reverse mapping of the unique identifiers to repetition addresses of specific inputs of the plurality of sets of input features is determined.

8. The method according to claim 6, characterized in that in: The repeated address is the specific address of the memory location where the input to the neural network layer is stored. The specific address is common between corresponding data slices of each processor in the plurality of processors, and The specific address is repeated between (i) a set of input features, or (ii) samples of the input features.

9. The method according to claim 2, characterized in that in: assigning each address partition to a data slice corresponding to an activation value or a gain value of a vector element of the multidimensional sparse tensor; and Each data slice of activation values ​​or gain values ​​is assigned to a specific processor for multiplication with the corresponding weight vector to perform the neural network calculation.

10. The method according to claim 1, characterized in that Generating the embedding output of the neural network layer includes: providing a plurality of inputs from a remote processor to a requesting processor, the plurality of inputs being stored in memory locations of a data slice assigned to the remote processor core; and A lookup is performed to obtain a corresponding weight vector, the weight vector including corresponding weight values ​​for multiplying an input of the plurality of inputs to generate a partial activation for a subsequent neural network layer.

11. The method according to claim 10, characterized in that in: A portion of the plurality of inputs is received from the memory location identified by a non-duplicate address; or from a memory location identified by an address for which there is no duplicate address in the original list.

12. A system for implementing a neural network on a hardware circuit to perform neural network calculations, characterized in that: The system comprises: one or more processing devices; and One or more non-transitory machine-readable storage devices for storing instructions executable by the one or more processing devices to perform operations including: A host receives a batch of inputs to a neural network layer, wherein each input in the batch of inputs is stored in a memory location identified by an address; identifying one or more duplicate addresses in a list of addresses for one or more inputs in the batch input; For each duplicate address: generating a unique identifier, the unique identifier identifying the duplicate address in the address list; For the batch of inputs, obtaining a first input from a memory location identified by an address corresponding to the unique identifier; An embedding output of the neural network layer is generated from the obtained first input.

13. The system according to claim 12, wherein: The neural network layer is an embedding layer of the neural network, the embedding output comprises an embedding feature vector, and the operations further comprise: The system's dispersal circuitry determines a reverse mapping of the unique identifier to a repetition address of a particular input in the batch of inputs to the embedding layer; and Based on the reverse mapping of the unique identifier to the repeated address of the specific input, one or more vectors are generated as outputs of the embedding layer.

14. The system according to claim 13, wherein: The system further comprises: generating a filtered list of addresses for the batch input based on one or more unique identifiers; determining, for addresses in the filtered address list, an address partition between each of a plurality of processors; and Wherein obtaining the first input comprises: for each address partition, using a corresponding processor assigned to retrieve input from the address, obtaining the first input from a memory location identified by the address in the address partition.

15. The system according to claim 13, wherein: Generating the embedding output of the neural network layer based on the reverse mapping includes: The address list including the duplicate address is parsed to map the embedded feature vector corresponding to the unique identifier generated for the duplicate address back to the specific input in the batch of inputs to the neural network layer.

16. The system according to claim 13, wherein: The system further comprises: The request processor provides the remote processor with an address of the first input, the address of the first input corresponding to the unique identifier and the request for the first input; The request processor receives the first input from the remote processor, the first input being stored in a memory location of a data slice assigned to the remote processor; and One or more reduction operations are performed to generate the embedding output of the neural network layer from a first input obtained from the data slice.

17. The system according to claim 13, wherein: The address list is used to form a batch input of input feature samples including multiple groups of input features, and generating the unique identifier includes: The unique identifier is generated for each repeated address on a corresponding set of input features in the input feature sample.

18. The system according to claim 17, wherein: Determining a reverse mapping of the unique identifier to the duplicate address comprises, The reverse mapping of the unique identifiers to repetition addresses of specific inputs of the plurality of sets of input features is determined.

19. The system according to claim 17, wherein: in: The repeated address is the specific address of the memory location where the input to the neural network layer is stored. The specific address is common between corresponding data slices of each processor in the plurality of processors, and The specific address is repeated between (i) a set of input features, or (ii) samples of the input features.

20. The system according to claim 13, wherein: in: assigning each address partition to a data slice corresponding to an activation value or a gain value of a vector element of the multidimensional sparse tensor; and Each data slice of activation values ​​or gain values ​​is assigned to a specific processor for multiplication with the corresponding weight vector to perform the neural network calculation.

21. The system according to claim 12, wherein: Generating the embedding output of the neural network layer includes: providing a plurality of inputs from a remote processor to a requesting processor, the plurality of inputs being stored in memory locations of a data slice assigned to the remote processor core; and A lookup is performed to obtain a corresponding weight vector comprising corresponding weight values ​​for multiplying inputs of the plurality of inputs to generate a partial activation for a subsequent neural network layer.

22. One or more non-transitory machine-readable storage devices, characterized in that For implementing a neural network on hardware circuitry to perform neural network computations, and for storing instructions executable by one or more processing devices to perform operations including: A host receives a batch of inputs to a neural network layer, wherein each input in the batch of inputs is stored in a memory location identified by an address; identifying one or more duplicate addresses in a list of addresses for one or more inputs in the batch input; For each duplicate address: generating a unique identifier, the unique identifier identifying the duplicate address in the address list; For the batch of inputs, obtaining a first input from a memory location identified by an address corresponding to the unique identifier; An embedding output of the neural network layer is generated from the obtained first input.

Images