Accelerated embedding layer computations

Abstract

Methods, systems, devices, and computer-readable media are described for performing neural network computations using a system configured to implement a neural network on hardware circuitry. The system includes a host receiving a batch input to a neural network layer. Each input is stored in a memory location identified by an address. The system identifies one or more duplicate addresses in the one or more input address lists. For each duplicate address, the system generates a unique identifier identifying the duplicate address in the address list. The system (i) obtains a first input from a memory location identified by an address corresponding to the unique identifier; and (i i) generating an output of the layer from the obtained first input.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511287322.3 (22) Application Date 2020.09.30 (30) Priority Data 62 / 909,667 2019.10.02 US (62) Divisional Application Data 202080060592.X 2020.09.30 (71) Applicant Google LLC Address California, USA (72) Inventors Rahul Nagarajjan, George Kurian, Homer Hariharan (74) Patent Agency Shanghai Huacheng Intellectual Property Agency Co., Ltd. 31300 Patent Attorney Xiao Hua (51) Int.Cl. G06N 3 / 063 (2023.01) G06N 3 / 045 (2023.01) G06N 3 / 0464(2023.01) G06N 3 / 0495(2023.01) G06N 3 / 084(2023.01) G06F 9 / 50(2006.01) (54) Invention Title: Accelerating Embedded Layer Computation (57) Abstract: A method, system, apparatus, and computer-readable medium are described for performing neural network computation using a system configured to implement a neural network on hardware circuitry. The system includes a host receiving batch inputs to a neural network layer. Each input is stored in a memory location identified by an address. The system identifies one or more duplicate addresses in a list of addresses of one or more inputs. For each duplicate address, the system generates a unique identifier that identifies the duplicate address in the list of addresses. The system (i) obtains a first input from the memory location identified by the address corresponding to the unique identifier; and (ii) generates the output of the layer from the obtained first input. Claims 1 page, Description 17 pages, Drawings 9 pages, CN 121168543 A 2025.12.19 CN 1 21 16 85 43 A 1. A method for performing neural network computation, characterized in that it comprises: identifying a set of addresses in memory in a request from a first processor; the first processor performing a deduplication operation on the set of addresses by: identifying duplicate addresses in the set of addresses; generating a unique identifier mapped to the duplicate addresses; and filtering the duplicate addresses from the set according to the unique identifier; after filtering the duplicate addresses from the set, a second processor retrieving data from the memory location identified by the address in the set of addresses using the unique identifier; and generating an output in response to processing the data using the second processor. 2. The method according to claim 1, characterized in that,The data includes: i) an input feature set and ii) a vector representing the values ​​of trainable weights of a neural network layer; and the unique identifier is used to store duplicate addresses of memory locations containing feature values ​​of the input features in the set. 3. The method of claim 2, wherein identifying duplicate addresses in the address set includes: identifying two or more input features having the same feature value in the input feature set. 4. The method of claim 2, wherein the vector of values ​​is an embedding of an embedding table stored on multiple data slices of the memory. 5. The method of claim 4, wherein the neural network layer is an embedding layer of an artificial neural network, and processing the data using the second processor includes: providing the feature values ​​of the data as input to the embedding layer; and processing the data in the embedding layer of the neural network. 6. The method of claim 5, wherein generating the output includes generating one or more embedded feature vectors as the output of the embedding layer. 7. The method of claim 6, wherein generating the unique identifier includes: generating a hash ID based on a memory offset of one of the duplicate addresses; and generating the unique identifier based on the hash ID. 8. The method of claim 1, wherein the method is performed using an integrated circuit comprising a distributed processor set, the distributed processor set comprising the first processor and the second processor. 9. The method of claim 8, wherein performing the deduplication operation comprises: filtering out duplicate addresses in the address set to alleviate load imbalance between the first processor and the second processor. 10. The method of claim 8, wherein performing the deduplication operation comprises: filtering out duplicate addresses in the address set to reduce the overall computational load of at least the second processor in the distributed processor set. 11. A non-transitory machine-readable storage device, wherein it is used to store instructions executable by a processing device to perform the method of any one of claims 1-10. 12. A system for performing neural network computation, comprising: a plurality of processors; and a non-transitory machine-readable medium for storing instructions executable by one or more of the plurality of processors to perform the method of any one of claims 1-10. Claims 1 / 1 page 2 CN 121168543 A Accelerating Embedded Layer Computing This application is a divisional application of the following application: International filing date of the original application: September 30, 2020; International application number of the original application: PCT / US2020 / 053442; National application number of the original application: 202080060592.XOriginal Application Title: Accelerating Embedded Layer Computation Cross-Reference to Related Applications

[0001] This application claims priority to U.S. Application 62 / 909,667, filed October 2, 2019, which is incorporated herein by reference. Background Art

[0002] This specification generally relates to using circuits to perform neural network computations.

[0003] A neural network is a machine learning model that employs one or more layers of nodes to generate outputs, such as classification, for received inputs. In addition to output layers, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to one or more other layers in the network, such as other hidden layers or output layers of the network. Some layers of the network produce outputs from received inputs based on the current values ​​of the corresponding set of parameters.

[0004] Some neural networks are convolutional neural networks (CNNs) (e.g., for image processing) or recurrent neural networks (RNNs) (e.g., for speech and language processing). Each of these neural networks includes a set of corresponding convolutional or recurrent neural network layers. A neural network layer may have an associated set of kernels and an embedding layer for processing inputs to generate a set of vectors for training the neural network. Kernels may be represented as tensors of weights, i.e., multidimensional arrays. For example, the embedding layer may process a set of inputs, such as image pixel data or activation values ​​generated by the neural network layer. The set of inputs or activation values ​​may also be represented as tensors.

[0005] This document describes techniques for accelerating computation of embedding layers in artificial neural networks. These techniques can be used in computing systems, such as large-scale distributed systems, which include circuitry configured to perform a deduplication operation on a list of addresses. During processing of inputs at the embedding layer of the neural network, such as during an example forward pass compute of the embedding layer, the deduplication operation is performed to reduce load imbalance on the distributed system.

[0006] This specification describes methods, systems, and apparatus for performing neural network computations using a system configured to implement a neural network on hardware circuitry, as well as a computer-readable medium. The system includes a host that receives batches of inputs to a neural network layer. Each input is stored in a memory location identified by an address. The system identifies one or more duplicate addresses in a list of one or more input addresses. For each duplicate address, the system generates a unique identifier to identify the duplicate address in the address list. The system (i) obtains a first input from the memory location identified by the address corresponding to the unique identifier; and (ii) generates the layer's output from the obtained first input.

[0007] One aspect of the subject matter described in this specification can be embodied as a method that uses a system configured to implement a neural network on hardware circuitry to perform neural network computation. The method includes: a host receiving a batch of inputs from a neural network layer specification (page 1 / 17, CN 121168543 A), wherein each input in the batch is stored in a memory location identified by an address; identifying one or more duplicate addresses in a list of addresses for one or more inputs in the batch. For each duplicate address, the method includes: generating a unique identifier that identifies the duplicate address in the list of addresses; for the batch of inputs, obtaining a first input from a memory location identified by an address corresponding to the unique identifier; and generating an embedded output of the neural network layer from the obtained first input.

[0008] Each of these and other embodiments may optionally include one or more of the following features. For example, in some embodiments, the neural network layer is an embedding layer of the neural network, the embedding output includes an embedding feature vector, and the method further includes a system distributed circuitry determining a reverse mapping of the unique identifier to a duplicate address of a specific input in the batch input of the embedding layer; and generating one or more embedding vectors as the output of the embedding layer based on the reverse mapping of the unique identifier to the duplicate address of the specific input.

[0009] In some embodiments, the method includes: generating a filtered list of addresses of the batch input based on one or more unique identifiers; determining an address partition among each of a plurality of processors for addresses in the filtered address list; and wherein obtaining the first input includes: for each address partition, obtaining the first input from a memory location identified by an address in the address partition using a corresponding processor assigned to retrieve input from the address partition.

[0010] In some embodiments, generating the embedding output of the neural network layer based on the reverse mapping includes: parsing the address list including the duplicate addresses to map the embedding feature vectors generated for the duplicate addresses, corresponding to the unique identifiers, back to the specific input in the batch input of the neural network layer.

[0011] The method further includes: a request processor providing a remote processor with an address of a first input corresponding to a unique identifier and a request for the first input; the request processor receiving the first input from the remote processor, the first input being stored in a memory location of a data slice allocated to the remote processor; and performing one or more reduction operations to generate the embedded output of the neural network layer from the first input obtained from the data slice.

[0012] In some embodiments, the address list is used to form an input feature sample comprising multiple sets of input features.Batch input and generating the unique identifier includes: generating the unique identifier for each duplicate address on each corresponding group of input features in the input feature samples. In some embodiments, determining the inverse mapping of the unique identifier to the duplicate address includes: determining the inverse mapping of the unique identifier to the duplicate address of a specific input of the plurality of input features.

[0013] In some examples, each address partition is assigned to a data slice corresponding to the activation or gain value of a vector element of a multidimensional sparse tensor; and each data slice of activation or gain value is assigned to a specific processor for multiplying with a corresponding weight vector to perform the neural network computation. In some embodiments, the duplicate address is a specific address of the memory location of the input stored in the neural network layer, the specific address being common among the corresponding data slices of each of the plurality of processors, and the specific address being repeated between (i) a set of input features or (ii) the input feature samples.

[0014] Generating the embedded output of the neural network layer includes: providing a plurality of inputs from a remote processor to a requesting processor, the plurality of inputs being stored in a memory location of a data slice allocated to the remote processor core; and performing a lookup to obtain a corresponding weight vector, the weight vector including a corresponding weight value for multiplying with an input of the plurality of inputs to activate a subsequent portion of the neural network layer generation. In some embodiments, a portion of the plurality of inputs is received from the memory location identified by an address in an original list of addresses where no duplicate addresses exist, or from a memory location identified by an address in an original list where no duplicate addresses exist. Specification 2 / 17 Page 4 CN 121168543 A

[0015] Other embodiments and aspects thereof include corresponding systems, apparatuses, and computer programs configured to perform methods encoded on a computer storage device. A system of one or more computers may be configured by means of software, firmware, hardware, or a combination thereof installed on the system to cause the system to perform actions in operation. One or more computer programs may be configured to have instructions that, when executed by a data processing device, cause the device to perform the actions.

[0016] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. A computing system includes circuitry that can be used to implement a processing scheme for accelerating the processing of an embedding layer for multiple batches of input. The circuitry and processing scheme can be used to reduce or mitigate load imbalance on a multi-core processing unit, such as a distributed system of processors, by filtering out duplicate addresses in a list of addresses used to retrieve inputs to an embedding layer of a neural network.

[0017] Having multiple duplicate addresses can be particularly problematic when computing or updating the embedding of a neural network. For example,Samples in a batch of inputs may include a variable number of features that need to be embedded, such as in the feature space of the embedding layer of a neural network. Each feature (or input) value has an embedding, such as a vector representing the values ​​of the trainable weights of the neural network stored in the embedding table of the neural network. While each value in a sample (or batch) has an embedding, different samples in a batch (and even different features within the same sample) may include the same feature value. In some cases, even the same feature within a sample may include the same feature value.

[0018] For example, when calculating or updating embeddings, the same feature value may need to be extracted for the same embedding table but for different features in a sample or batch, or for various features in different samples or batches of input. Because the same input value may need to be extracted multiple times (e.g., from the same address in memory), methods for filtering out duplicate addresses when calculating or updating embeddings can reduce the computational load on a distributed system and help improve or accelerate the computation of the embedding layer of a neural network.

[0019] Deduplication techniques may be performed at a source or requesting processor before addresses in a list are broadcast to different remote processors in a distributed system assigned to retrieve specific data elements of a given batch of input for processing at the embedding layer. The techniques described herein can be used to more efficiently compute the embedding output of a neural network layer from retrieved data elements as input for a batch. For example, the process of generating the embedding output may include retrieving the embeddings of one or more inputs in the batch, and performing reduction on the embeddings of relevant input features in the batch.

[0020] Compared to existing systems, the techniques described herein can be used to reduce or minimize the computational load performed to retrieve embedding data, perform reduction, and return a reduced vector of embedding values. When generating the embedding layer output, this technique provides an improved method for balancing the computational load between a request processor and a remote processor, whereby the request processor processes the reduced embedding values ​​of the batch inputs and the remote processor retrieves the embeddings corresponding to the batch inputs.

[0021] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of this subject matter will become apparent from the embodiments, the drawings, and the claims. Brief Description of the Drawings

[0022] FIG1 is a block diagram of an example computing system.

[0023] FIG2 is a block diagram of an example circuit including an example multi-core processing unit.

[0024] FIG3 illustrates an example host broadcasting a set of addresses to the processor of the example multi-core processing unit.

[0025] Figures 4A-4D illustrate diagrams associated with examples of processing schemes involving two or more processors of an example multi-core processing unit. Specification 3 / 17 page 5 CN 121168543 A

[0026] Figure 5 illustrates an exemplary processor of an exemplary multi-core processing unit.

[0027] Figure 6 is a flowchart illustrating an exemplary process for generating the output of a neural network layer.

[0028] The same reference numerals and designations in the figures denote the same elements. Detailed Description

[0029] This specification describes techniques for accelerating embedding layer computation to mitigate processing latency that may occur due to processor load imbalance in a distributed computing system. For example, latency can occur during processor computation to generate the output of an embedding layer of a multi-layer neural network. Specifically, when a particular processor or core of a distributed system is required to perform a large number of query and computation operations (e.g., reducing or concatenating retrieved values) to produce embeddings, that particular processor may experience processing latency corresponding to load imbalance.

[0030] Imbalance can exist between a source processor that submits a request for data to query or retrieve input values ​​and a remote processor that performs a lookup to retrieve the data. The data may be embeddings (e.g., weights) of an embedding table, and the remote processor may be tasked with processing the request to return the embedded data or weight values. For example, in a forward pass computation operation of an embedding layer, the remote processor processes a request from the source processor that causes the remote processor to look up an address to retrieve data stored in memory and perform computations using the retrieved data. In existing distributed architectures, remote processors perform a deduplication of the address specified in the request.

[0031] For example, in existing systems, a remote processor receives one or more duplicate addresses and performs data retrieval and associated simplification operations corresponding to the inputs of all received addresses (including duplicate addresses). However, the architectures and processing schemes used in some existing systems are prone to load imbalance problems, especially when the computational load at one or more remote processors is unbalanced with the computational load at the source processor, or when there is an imbalance between two or more remote processors. These load imbalances can cause computational operations that generate the output of the embedding layer to stop or be significantly delayed.

[0032] Typically, the embedding layer of a neural network is used to embed features into a feature / embedding space corresponding to the embedding layer. Features can be attributes or properties shared by independent units on which analysis or prediction is to be performed. For example, an independent unit can be a set of words in a vocabulary or image pixels that make up items such as images and other documents. Algorithms for training the embedding layer embeddings can be executed by the neural network processor to map features to embedding vectors. In some implementations, the embeddings of the embedding table are learned together with other layers of the neural network on which the embeddings are to be used. This type of learning occurs by updating the embedding table through backpropagation of gradients.

[0033] In other embodiments, the embeddings can be learned independently of other layers of the neural network to which the embeddings are to be used, for example, when the embeddings are pre-trained. For example, by processing information about discrete input features, the neural network processor can enable...The algorithm is used to compute embeddings to determine mappings or placements of similar inputs to embedding vectors that are geometrically close in the embedding space. In some cases, the process of computed embeddings can represent a technique of feature learning or feature engineering, which allows the system to automatically discover the representations needed for feature detection from raw input data.

[0034] In some implementations, a given “input” may have one or more features of one or more types, and the embedding layer generates a corresponding embedding for each of these types. For example, the input may be used for a search query with several different feature types. Feature types may include attributes of a user or user device (e.g., location, preferences, device type, etc.), query tokens, previously submitted queries, or other relevant types that may correspond to attributes of the search query. For any feature type with more than one feature for a given input, the computational system can be operated to retrieve individual embeddings of each of these features. The system can also be operated to combine the retrieved embeddings, for example, by computing the average of the embedding values, to generate a final embedding for that feature type. Specification 4 / 17 Page 6 CN 121168543 A

[0035] Based on the context of the above discussion, this specification describes techniques for implementing dedicated hardware circuitry configured to perform address deduplication at a source processor before sending an address list to a remote processor for lookup in a portion of memory allocated to the remote processor. The hardware circuitry includes an identifier (ID) scatter circuit (“scatter circuit”) configured to parse the original address list to filter out one or more “duplicate” addresses. The scatter circuit is configured to filter duplicate addresses before bucketing addresses belonging to each remote processor.

[0036] For example, a system including the hardware circuitry can generate instructions to cause the scatter circuit to bucket addresses belonging to a data shard allocated to a specific remote processor of the system. The system then sends the addresses to the specific remote processor allocated to the data shard, which stores an embedding vector corresponding to the input (feature) stored at the address. The embedding vector may be a weight value or a portion of an embedding table stored on the data shard. In some cases, the embedding vector of the embedding table can be partitioned across multiple remote processors (processor nodes or cores) of the computing system. The operable host system sends a set of addresses or vector indices to a subset of the remote processors to perform operations associated with retrieving weight vectors, such as the embedding vectors, which are mapped to specific features stored at addresses in the set. In some examples, the remote processors perform one or more query and computation operations before providing or returning the weight vectors to the system host.

[0037] The hardware circuitry includes a reverse mapper configured to augment or modify addresses by adding metadata, such as a hash key, to the addresses. The metadata is used to reverse map a specific set of retrieved or computed data (e.g., collected vectors) back to their corresponding original addresses before filtering out duplicate addresses. In some cases, the circuitry is configured to perform sparse reduction on data received from a remote core to form a dense matrix.

[0038] For example, before deduplication, the circuitry is augmented to iterate through each address in the original list of address locations and collects vector outputs corresponding to specific input addresses using metadata tags generated by the reverse mapper, and performs reduction on the collected vectors. These techniques allow computations in a distributed system of processors, with multiple batches of input, to be performed by a specific master processor, rather than by a remote processor mapped to a specific portion of memory that holds weight values ​​(e.g., parameters) corresponding to one or more inputs in the batch.

[0039] Figure 1 shows a block diagram of an example computing system 100 configured to retrieve data elements stored in the memory of system 100 for performing embedding layer operations to generate an embedding set for training a neural network. In some cases, training the neural network may include determining one or more learning objectives based on an output vector representing a neural network embedding. For example, the embedding output may correspond to one or more sets of output features, which include floating-point / parameter vectors of a given output dimension (64-d, 256-d, etc.).

[0040] When the neural network of system 100 is trained to perform certain computational functions, such as image or speech recognition, embedding outputs are generated. In some embodiments, training the neural network includes updating the embedding set previously stored in the embedding table of the neural network, for example, during a prior phase of training the neural network. In other words, the embeddings of the neural network may be pre-trained. For example, the embeddings of the embedding layer of the neural network may not be trained together with the neural network to which the embeddings are to be used. Therefore, the techniques described herein can be used to update the embeddings of a pre-trained neural network, improving efficiency compared to existing methods, such as those used for pre-training embeddings.

[0041] The computing system 100 includes a host 102, a multi-core processing unit 104, and data shards 106a-106k, where k is an integer greater than 1. Typically, the host 102 may be a processing unit, such as a processor, multiple processors, or multiple processor cores. Therefore, the host 102 may include one or more processors and is operable to generate or process instructions for accessing the target dense matrix and to send instructions 110 to the multi-core processing unit 104 to generate the target dense matrix. As described in more detail below.As described on page 7 of document 5 / 17, CN 121168543 A, performing embedding layer operations may include transforming sparse elements from one or more matrices to generate a dense matrix.

[0042] Multi-core processing unit 104 accesses corresponding elements 108a-108n from one or more data shards 106a-106k, where n is an integer greater than 1. Multi-core processing unit 104 uses the corresponding elements 108a-108n to generate a target dense matrix 112 and provides the target dense matrix 112 to host 102 for further processing. Elements 108a-108n may be two-dimensional matrices of different sizes, and multi-core processing unit 104 may generate the target dense matrix 112 by transforming each of elements 108a-108n into a vector and concatenating n vectors into a single vector.

[0043] Typically, in the case of embedding, the “sparse” information corresponding to the sparse elements may be a one-hot vector identifying the feature values. For example, if there are five possible values ​​for a given feature (e.g., A, B, C, D, E), the sparse vector identifies the feature value “A” as (1, 0, 0, 0, 0, 0), and the embedding layer maps (1, 0, 0, 0, 0) to a dense embedding vector for the feature value “A”. In some implementations, during training the embedding layer to learn the embedding, elements 108a-108n can be weight values ​​of an embedding table that is converted into vectors, such as embedding vectors for feature values ​​“B” or “C”. The weight values ​​can be transformed using a neural network processor of multi-core processing unit 104, which executes a training algorithm to compute the embeddings based at least on the feature-to-embedded vector mapping.

[0044] Host 102 can process instructions to update a target dense matrix and send the updated dense matrix to multi-core processing unit 104. For example, the target dense matrix can correspond to the embedding of a pre-trained neural network. Therefore, host 102 can process instructions to update the embedding of the pre-trained neural network, thereby generating the updated dense matrix. For example, during subsequent iterations of training the neural network to update the embeddings, a backward pass can be performed to update the embeddings by determining a new mapping from the input features to the embedding vectors and generating an updated dense matrix based on that new mapping. In some implementations, the multi-core processing unit 104 is operable to convert the updated dense matrix into corresponding sparse elements and update one or more sparse elements (e.g., weights) stored in data shards 106a-106k accordingly.

[0045] As described above, the host 102 is configured to process instructions executed within the computing system 100. In some implementationsIn this embodiment, host 102 is configured to process a target dense matrix 112 generated by multi-core processing unit 104. In some other embodiments, host 102 may be configured to request multi-core processing unit 104 to generate target dense matrix 112, and another processing unit may be configured to process target dense matrix 112.

[0046] Each processor of multi-core processing unit 104 is configured to retrieve data elements stored in memory of system 100. Memory may include multiple data fragments 106a-106k storing data including elements 108a-108n. Data may include inputs, activations, gain values, or weight values ​​corresponding to parameters or kernels of the weight matrix structure. In some embodiments, data fragments 106a-106k may be volatile memory cells or multiple cells. In some embodiments, data fragments 106a-106k may be non-volatile memory cells or multiple cells. Data fragments 106a-106k may also be another form of computer-readable medium, such as a device in a storage area network or other configuration. Data shards 106a-106k can be coupled to the multi-core processing unit 104 using electrical, optical, or wireless connections. In some embodiments, data shards 106a-106k may be part of the multi-core processing unit 104 and are based on a processor-in-memory (PIM) architecture.

[0047] The multi-core processing unit 104 is configured to determine a dense matrix based on sparse elements. In some embodiments, the multi-core processing unit 104 may be configured to determine the location of sparse elements based on a dense matrix. The multi-core processing unit 104 includes a plurality of interconnected processors or processor cores. For example, the multi-core processing unit 104 may be a distributed processing system including a plurality of interconnected processor cores. Generally, the terms “processor” and “processor core” are used interchangeably to describe the discrete interconnected processing resources of the multi-core processing unit 104.

[0048] FIG2 is a block diagram of an exemplary dedicated circuit 200 including an example multi-core processing unit 104. Typically, a computing system 100 can be implemented using one or more circuits 200 (described below) as per the specification page 6 / 17, 8 CN 121168543 A. In the embodiment of FIG2, the multi-core processing unit 104 has a set of interconnected processors including processors 0 to 15. As described in more detail below, each of the processors 0-15 of the multi-core processing unit 104 can be assigned to one or more of the data slices 106a-106k to obtain data (e.g., feature values ​​and weights) stored in the memory locations of the data slices assigned to the processors. Each processor of the multi-core processing unit 104 is operable to receive requests from the host 102 or from another processor in the multi-core processing unit 104.The identified memory location is fetched or retrieved.

[0049] The processors cooperate to retrieve data elements mapped to the embedding table. The data elements are retrieved to perform embedding layer operations, such as computations to generate output vectors (e.g., dense vectors) representing the embeddings of the neural network. In some implementations, the embedding output is a mapping of discrete inputs in a batch of inputs to dense vectors of numerical values ​​(e.g., floating-point values), which can be used to define relationships between objects such as terms or words in a vocabulary. For one input in a given batch, the embedding can be a vector, while for the entire batch of inputs, the embedding can be a matrix, such as a set of vectors. In some implementations, the vector of embedding values ​​corresponds to trainable weights learned by the neural network during training. The learned weights can encode one or more words by looking up dense vectors stored in the embedding table, where the dense vectors correspond to specific words. Multiple dense vectors can correspond to dense matrices used to train the neural network to perform a specific task or function related to recognizing sequences of words, such as language processing.

[0050] In some examples, embedding layer computation is a large-scale data lookup problem typically used during the training phase of a neural network. For example, during training, a large table of weight values ​​can be sharded or partitioned across multiple dimensions of the multidimensional tensor, such as the x, y, or z dimensions of the multidimensional tensor. Specifically, the table of weight values ​​can be sharded across data shards 106a-106k such that each weight value is stored at a corresponding location in the memory including the data shards.

[0051] Furthermore, one or more of the data shards 106a-106k can be partitioned into specific processors 0-15 in the multi-core processing unit 104. Each shard or partition of the table corresponds to a data shard 106k allocated to a specific processor or processor core of the multi-core processing unit 104. In some examples, each processor core is a corresponding neural network processor. In other examples, each processor core is a corresponding core of a single neural network processor. Typically, the computing system 100 may include multiple processors or multiple processor cores based on the desired configuration of the system.

[0052] Referring again to FIG. 2 and the hardware circuit 200, the circuit includes an identifier (ID) scattering circuit 202 (“scattering circuit 202”), which includes a hash generator 204 and a reverse mapper 206. Each of the hash generator 204 and the reverse mapper 206 is described below.

[0053] The scattering circuit 202 is configured to process a list of raw addresses that can be received by the host 102. The list of raw addresses identifies memory locations in the memory of the system 100, such as memory including data fragments 106a-106k.In Figure 2, the distributed circuit 202 is shown as being external to or separate from the host 102 and the multi-core processing unit 104. However, in some embodiments, the distributed circuit 202 may be part of the host 102, such as a distributed unit or computing module, or a sub-circuit of the host 102.

[0054] Similarly, in some embodiments, the distributed circuit 202 may be included in the multi-core processing unit 104 as part of a processor in the multi-core processing unit 104. For example, the distributed circuit 202 may be included in the source / main processor (e.g., processor 0) of the multi-core processing unit 104. The source processor may be tasked by the host 102 to generate one or more requests and provide the requests to a plurality of other processors (e.g., processors 1-15) in the multi-core processing unit 104 to retrieve data elements from memory locations of data shards 106a-106k allocated to these processors.

[0055] The distributed circuit 202 communicates with the host 102 to receive a raw address list. For example, host 102 receives batch of inputs to layer A of the neural network specification (page 7 / 17, CN 121168543 A), such as from an external source or a controller of system 100 that manages machine learning operations performed by the system. For example, machine learning operations can be used to train the neural network to process image or audio data, enabling the neural network to learn how to identify specific objects in an image / video or how to detect certain word sequences in an audio stream.

[0056] Host 102 receives one or more batches of input. In some embodiments, host 102 receives multiple batches of input corresponding to one or more samples. For example, a first set of multiple batches of input may correspond to a sample 210-1, and a second set of multiple batches of input may correspond to another sample 210-2. Each sample 210-1, 210-2 may include input corresponding to a set of features of an object (e.g., an image or audio stream). For example, each of samples 210-1 and 210-2 may include a corresponding batch of features that correspond to features of different regions of the same image, or to different regions of multiple different images.

[0057] Each input in a batch is stored in a memory location, and this location may be identified by an address (e.g.,

[99] ,

[736] ,

[37] ,

[99] , etc.) in a raw list of addresses received by the distributed circuitry 202 from the host 102. In some embodiments, each of the inputs has a corresponding set of weights, which may be stored as data or sparse elements of an embedding table, which is fragmented in data shards 106a-106k and partitioned in processors 0 to 15. In some embodiments, the embeddings of the individual features in the batch inputs are stored in a distributed manner, for example, across various data shards of system 100.

[0058] For example, features may be distributed across multiple processor nodes or cores (0-15), with each node allocated a memory location for storing the location of the embedded fragment. As described in more detail below, host 102 (e.g., a source processor) provides a request to one or more of processors 0-15 (e.g., remote processors) including one or more subsets of addresses for processing to retrieve data elements for generating a dense matrix.

[0059] As described above, in some existing systems, the remote processor receiving duplicate addresses performs data retrieval, arithmetic operations (e.g., multiplication), and associated reduction operations on the received addresses, including duplicate addresses. This can lead to load imbalance problems, particularly when the computational load at the remote processor is unbalanced with the computational load at the requesting or source processor.

[0060] The distributed circuitry is configured to parse the original address list to filter out one or more “duplicate” addresses. The address list received by host 102 includes one or more duplicate addresses corresponding to the input features in the batch. For example, the raw address list received at system 100 may include addresses of the batch inputs

[26] ,

[96] ,

[99] , [1], [7],

[312] ,

[99] , and

[912] . In this example, the raw address list includes address

[99] as a duplicate address in the list. The scattering circuit 202 is configured to filter duplicate addresses before bucketing the addresses belonging to each remote processor and providing a request to the remote processor to retrieve the data element corresponding to the group address.

[0061] The scattering circuit 202 identifies one or more duplicate addresses in the address list for each input in the batch input. For example, a single embedding table may be referenced by multiple features. Thus, duplicate addresses may exist on multiple different features. For example, the embedding table T may be referenced by features F1 and F2. In this example, duplicate addresses (e.g.,

[99] ) may exist within the input of F1 (in one or more samples) and within the input of F2 (in one or more samples).

[0062] The scattering circuit 202 is configured to perform deduplication on the original address by filtering out duplicate addresses and providing unique addresses (identifiers) to the multi-core processing unit 104 for processing at one or more of the processors 0-15. For each duplicate address: the scattering circuit 202 generates an identifier that uniquely identifies the duplicate address in the original address list. Then, based on certain identifiers indicating duplicate addresses in the original address list, the scattering circuit 202 filters the original address list to generate a filtered address list such that the filtered address list includes only unique identifiers corresponding to non-duplicate addresses.

[0063] More specifically, for each address in the original list, the scattering circuit 202 uses a hash generator 204 to generate a hash. (Page 8 / 17 of specification, 10 CN 121168543 A)An identifier corresponding to a hash ID. The memory of system 100 may have address values ​​with different memory offsets (e.g., offset 0, offset 1, etc.) (e.g.,

[99] ). Hash generator 204 may generate hash IDs based on address values, memory offsets of address values, or combinations of each. For example, for repeated occurrences of addresses in the list, a portion of the hash ID generated based on the address value (e.g.,

[99] ) will be repeated, but a portion of the hash ID will differ from a second portion of the hash ID generated based on the memory offset, or based on a combination of address values ​​and memory offsets. In some examples, scattering circuitry 202 may append the corresponding first portion of the hash ID to an address in the original list as a prefix to the address. scattering circuitry 202 may recursively scan the address list to identify repeated occurrences of the appended prefix and filter out the specific address based on such repeated occurrences.

[0064] In some embodiments, the appended prefix, or the first portion of the hash ID, is an example of a metadata tag used by scattering circuitry 202 to augment addresses in the original list to generate a filtered address list. Based on the unique identifiers generated by hash generator 204, scatter circuit 202 generates a filtered list of addresses for the batch input. The filtered list of addresses includes only unique identifiers or addresses (e.g., non-duplicate addresses) of the memory locations of data fragment 106. The filtered list of addresses is then provided to multi-core processing unit 104, such that processors 0-15 receive only unique addresses and not duplicate addresses that appear multiple times.

[0065] scatter circuit 202 uses reverse mapper 206 to generate a reverse map that maps the collected vectors back to specific addresses in the original list. Reverse mapper 206 is configured to add one or more metadata tags to the address or batch input to enable reverse mapping. For example, reverse mapper 206 can enhance or modify addresses in the original list by adding metadata to the addresses. Metadata can be metadata tags that include hash IDs or annotations specifying the memory offset of the address. Before filtering duplicate addresses, reverse mapper 206 uses metadata to reverse map a specific set of retrieved or computed data (e.g., collected vectors) to the corresponding original addresses.

[0066] In this manner, circuit 200 provides dedicated hardware operable to traverse each address in the original list of address locations before deduplication, and to collect vector outputs corresponding to specific input addresses using metadata tags generated by reverse mapper 206. As described in more detail below, circuit 200 is operable to perform a reduction operation on the collected vectors to generate a dense matrix. For example, an example source processor of host 102 or multi-core processing unit 104 may include exemplary sparse reduction circuitry.The reduce circuit is configured to perform the reduction of data elements received from a remote core to form a dense matrix. This will be described in more detail below with reference to FIG5.

[0067] FIG3 shows a host 102 broadcasting a set of addresses to each of the processors 0, 4, 8 and 12 of the multi-core processing unit 104. As shown in FIG3, the host 102 sends a unique identifier or address of the batch input to the processors of the multi-core processing unit 104. In some embodiments, the host 102 may send a unique identifier for a corresponding batch input corresponding to each of sample 210-1. For example, the host 102 may send the address [0, 26, 96, 99] of batch input 302 to processor 12 and the address [2, 9, 17, 736] of batch input 304 to processor 0. As shown in FIG2, the address of batch input 302 is used for sample 210-1, while the address of batch input 304 is used for sample 210-2.

[0068] In some embodiments, host 102 is operable to provide different batches of input to different processors and different column group processors in multi-core processing unit 104. For example, host 102 may send different batches of input to different processors in a sequential or parallel manner. Similarly, host 102 may send different batches of input to different column group processors in a sequential or parallel manner.

[0069] Each of Figures 4A-4D shows a corresponding diagram associated with an example of processing scheme 402, which may be implemented at system 100 using at least host 102 and two or more processors in multi-core processing unit 104.

[0070] As described above, processors 0-15 cooperate to retrieve data elements mapped to an embedded table, such as a sparse embedded table comprising multiple sparse elements. Data elements are retrieved in response to a request provided by a source (or requesting) processor processed by a remote processor. In some cases, data elements are retrieved to perform embedding layer operations, such as computations to generate output vectors (e.g., dense vectors) representing neural network embeddings, or computations to generate partial activations in response to multiplying inputs (e.g., gain values ​​or activation values) with weight vectors.

[0071] As described above, the embedding table may be a table of weight values ​​partitioned between data fragments 106a-106k, such that each weight value is stored in a corresponding location in memory. Data fragments 106a-106k may be partitioned into specific processors 0-15 in the multi-core processing unit 104, such that the weight values ​​in the embedding table correspond to the data fragments allocated to specific processors 0-15 of the multi-core processing unit 104.

[0072] As described below (and referring to FIG. 5), a request provided by a source processor to a remote processor to retrieve a data element may include an address (e.g.,

[912] ) identifying a memory location of a data fragment. The memory location stores input data, such as gain values ​​or activations. The request is provided to cause the remote processor to retrieve a data element representing an input, such as one of a plurality of inputs received at host 102. In some embodiments, the request is provided to cause the remote processor to perform a lookup operation to retrieve a specific or unique weight corresponding to a particular input in the batch of inputs.

[0073] In the example of FIG. 4A, block 402 illustrates an exemplary processing scheme in which at least processors 0-3 cooperate to retrieve a data element mapped to a portion of an embedded table. In this example, processor 0 may be a source (or requesting) processor that submits a corresponding request to each of processors 1, 2, and 3. More specifically, processor 0 is the source processor, and each of processors 1, 2, and 3 is a remote processor relative to processor 0.

[0074] In some cases, processors 0, 1, 2, and 3 form a subset 404. In subset 404, each of processor 0, processor 1, processor 2, and processor 3 can be a source processor relative to other processors in subset 404, wherein the other processors are remote processors that receive and process requests from source processors in subset 404. In some cases, source processors and remote processors in subset 404 may still receive and process requests from another processor in multi-core processing unit 104 or from host 102, wherein host 102 may be a master, source processor not included in the subset.

[0075] The exemplary scheme shown in block 402 may be part of a processing technique implemented at system 100 to generate the output of a neural network layer, such as the embedded output of an embedded neural network layer or the output of a first hidden layer of a neural network. The processing technique includes one or more remote processors that provide data elements comprising multiple inputs and corresponding query values ​​to a requesting processor. The data elements are stored in memory locations of data slices allocated to the remote processors.

[0076] In some embodiments, in response to each remote processor performing a lookup, the data elements are provided to the source processor to obtain a weight vector, the weight vector comprising corresponding weight values ​​multiplied by a specific input of the multiple inputs. For example, a portion of memory can be allocated to a remote processor to store the weights of the kernel matrix, which are multiplied by the activations at the corresponding addresses stored in a list of addresses received by the source processor. The input and weight multiplication, for example, generates partial activations in subsequent neural network layers (described with at least reference to Figure 4D below). Example operations will be described to further illustrate this processing technique.

[0077] During exemplary operation at system 100 processing multiple batches of input, a raw list of multiple addresses is received by host 102 or another source processor, such as processor 0 of multi-core processing unit 104. Address

[912] may be included in multiple addresses in the list. For a given batch or sample,

[912] may appear more than once in the received list. Specifically, address

[912] may be one of multiple addresses copied between (i) a set of input features or (ii) samples of input features. In some embodiments, each repetition of address

[912] in a batch or sample may be used for a specific memory offset in a given data slice.

[0078] Address

[912] may be a location in memory storing an input / feature value that is an identifier (e.g., a hot identifier) ​​of a word in a table in a given vocabulary specification 10 / 17 pages 12 CN 121168543 A, such as the word "car". For example, an embedding table may map the hot identifier of a word to an embedding. More specifically, the address

[912] may be stored as an instance of an input feature that appears multiple times within the feature values ​​of a given sample and / or within the input of a given batch. Each occurrence of the address

[912] may be a feature with an embedding, such as a specific weight or vector of trainable weight values ​​stored in an embedding table in memory.

[0079] System 100 may perform deduplication on the original list to remove multiple duplicate occurrences of the address

[912] , thereby generating a filtered address list that includes only unique addresses. Reverse mapper 206 may append metadata tags to specific occurrences of the address

[912] in a batch or sample to map the collected vectors (e.g., weight vectors) back to specific occurrences of the address

[912] in the original list. As described in more detail below, an example source processor may use this reverse mapping to perform computations locally, such as multiplying between (i) the input value stored at that specific occurrence of the address

[912] and (ii) the collected vectors received from a remote processor.

[0080] Typically, address

[912] may identify a memory location of a data slice allocated to processor 0, the memory location storing a specific input value. In some embodiments, address

[912] in a data slice allocated to processor 0 may also be repeated, for example, 100 times in each batch of multiple batches of input, but with reference to different data slices allocated to other processors of processing unit 104. For example, address

[912] may be a specific address common to the corresponding data slices allocated to one or more processors in multi-core processing unit 104.

[0081] For example, the example source processor of host 102 or processor 0 may perform 16 memory lookups for the multi-core processor of system 100.Each processor in core processing unit 104 performs a memory lookup once, but does not perform calculations related to reduction or matrix multiplication. In the example of FIG4A (and 4C discussed below), processor 3 may send only its corresponding input values ​​(e.g., activation or gain) and weight values ​​to processor 0, which correspond to unique addresses that can be generated for multiple repetitions of address

[912] in a particular batch.

[0082] In the example of FIG4B, remote processor 104n is configured to locally reduce its own lookup value based on instructions from the requesting processor. In some embodiments, each of the processors in multi-core processing unit 104 may represent a neural network processor chip that is assigned to look up and locally reduce data elements stored at specific data shards 106a-106k. The corresponding remote processor is configured to look up and locally reduce data elements in its embedding table based on instructions provided by host 102 or based on instructions received directly from a request / source processor comprising a subset of the corresponding remote processor.

[0083] For example, processor 3 may be a corresponding remote processor in a subset of processors including processor 0, processor 1, and processor 2. In this example, processor 0 may be a request processor in a subset including processor 3, and processor 3 is a remote processor at least relative to processor 0. In this way, processor 3 is configured to look up and locally reduce data elements, such as unique weight values ​​of the input stored at address

[912] , to generate an exemplary weight vector [f2].

[0084] In the example of FIG. 4C, the source or request processor (e.g., processor 0) receives locally reduced data elements at each remote processor and performs one or more operations on these data values. In some embodiments, the request processor is configured to perform additional reduction operations on the received values. For example, when processor 3 retrieves and locally reduces the input stored at address

[912] and the corresponding lookup data to generate the value of vector [f2], operable system 100 determines the reduction of the input relative to the corresponding sample or batch of input to which the input is assigned.

[0085] For example, using request processor 0 of subset 404, system 100 can determine that input

[912] needs to be reduced by a factor of five for sample 210-1 and by a factor of two for sample 210-2. Additionally, request processor 0 is configured to perform reduction on all data values ​​received from each remote processor in subset 404. For example, request processor 0 is configured to locally reduce the weight vector [f2], weight vector [g1], and weight vector [g2] received from remote processor 1, remote processor 2, and remote processor 3, respectively. (See page 11 / 17 of document 13 CN 121168543 A)Each of [f1]. In some implementations, request processor 0 is configured to look up its own data elements, such as unique weight values ​​of the input stored at addresses [1], [7], [2], [9], and

[17] , and locally reduce the lookup values ​​on these features to generate example weight vectors [f0] and [g0]. The request processor may perform reduction on its own lookup values ​​and data values ​​received from each remote processor in subset 404.

[0086] In the example of FIG. 4C, each of subsets 406 and 408 may be used to implement a processing scheme similar to or the same as the processing scheme 402 described above. For example, in subset 404, processor 7 is configured to look up and locally reduce the data elements of the remote processor, such as unique weight values ​​of the input stored at address

[37] , to generate an exemplary weight vector [e1]. In subset 406, a source or request processor (e.g., processor 4) receives data elements of the locally reduced vector [e1] at remote processor 7 and performs one or more operations on these data values. Then, processor 4 can generate exemplary weight vectors [d0] and [e0] based on one or more cascading or reduction operations performed on the embeddings retrieved for various input values ​​in a batch or sample.

[0087] In some embodiments, each of processors 1-15 can send its corresponding input value (e.g., activation or gain) and weight value to processor 0, the input value and weight value corresponding to a unique address that can be generated for multiple repetitions of address

[912] in a particular batch. In other embodiments, each of processors 0-15 can be assigned to process a corresponding batch, and each processor will perform its corresponding computation for 100 repetitions of address

[912] in its corresponding batch.

[0088] The processing techniques described in Figures 4A-4D allow a processing-specific master processor with multiple batches of input to perform computations in a distributed system of processors, rather than by a remote processor mapped to a specific portion of memory that holds weight values ​​(e.g., parameters) corresponding to one or more inputs in the batch. These techniques can also translate into improved load balancing and scalability of system 100.

[0089] In the example of FIG4D, processor 12 may represent a processing node of an example distributed processing system of system 100. In some embodiments, processor 12 may be a main processor having multiple batches of input. Processor 12 is configured to receive each of vector outputs (e.g., dense vectors), the vector outputs being reduced by a corresponding request processor 104n in the multi-core processing unit 104. More specifically, processor 12 may receive vector outputs corresponding to data elements of each output matrix (representing the embedding of a neural network), the vector outputs being reduced locally by a corresponding remote processor.

[0090] Processor 12 is configured to perform computations to generate partial activations in response to multiplying an input, such as a gain value or activation value, with the weight values ​​of a weight vector. For example, processor 12 is configured to generate one or more sets of partial activations based on the multiplication between the input and the weight values ​​of weight vectors [a] and [b].

[0091] FIG5 illustrates an example processor 104n of multi-core processing unit 104, where n is an integer greater than 1. For example, processor 104n may be any of the processors 0-15 described above with reference to FIG2. Generally, processor 104n is configured to receive a request 542 from node network 520 to extract or retrieve data (e.g., sparse elements) stored in one or more data shards 530.

[0092] For example, for a dense matrix generated using sparse elements, source processor 502 sends request 542 to processor 104n in node network 520. Node network 520 may be a two-dimensional mesh network. In some implementations, the data extraction request 542 includes instructions to transform the extracted sparse elements into a dense matrix. As described above, the request to transform the extracted sparse elements into a dense matrix can be executed during training the embedding layer to learn the embedding. For example, the extracted elements may be weight values ​​of an embedding table, which are transformed into embedding vectors based at least on the mapping from input features to embedding vectors. The source processor 502 may broadcast the request 542 to the processor 104n. The routing of the broadcast request 542 may be similar to the description of FIG. 2 of a host 102 providing a request to one or more processors 104n. In some implementations, the source processor 502 may be the host 102 or another processor 104n included in the multi-core processing unit 104.

[0093] Generally, processor 104n is configured to receive request 542 to retrieve a data element stored in one or more data fragments 530, and is configured to determine whether processor 104n is assigned to access the address of a location in memory where the data indicated by request 542 is stored. In some embodiments, processor 104n may determine whether it is assigned to access the data element indicated by request 542 by using a lookup table. For example, if the address of the specifically requested data element (e.g., the address at memory offset 1 of lookup table No. 5

[99] ) is included in the lookup table, processor 104n may signal to an example data retrieval unit of the processor to retrieve the specifically requested data element. In some embodiments, processor 104n may be configured to broadcast the received request 542 to another processor 104n on node network 520.

[0094] Processor 104n is configured to retrieve one or more requested data elements from a specific data slice 530 allocated to processor 104n in response to processing request 542. In some embodiments, processor 104n is a vector processor, vector processing unit (VPU), array processing unit, or any suitable data computing resource for processing arrays and matrices of vector elements. In some embodiments, each of the processors 104n included in multi-core processing unit 104 may be allocated to a specific data element stored in data slice 530, such as a specific data element of an embedded table, based on the address used to store the data element and the corresponding memory location.

[0095] In some embodiments, source processor 502 may generate one or more requests 542 for multiple batches of input data elements to be retrieved and processed using one or more processors 104n. In some cases, each batch of input may be associated with a corresponding request 542 for a subset of the requested data elements. Each processor 104n is configured to independently retrieve assigned data elements from its assigned data slice 530. In some implementations, as described in more detail below, processor 104n causes the retrieved data to be forwarded to one or more units of the processor for further processing.

[0096] As shown in FIG5, processor 104n includes a sparsity reduction unit 506, a cascade unit 508, and a compression / decompression unit 510. Processor 104n may forward retrieved data elements to sparsity reduction unit 506. Sparsity reduction unit 506 is configured to reduce the dimension of the retrieved data elements. For example, processor 104n may generate vector elements with a dimension of 100×1. Sparsity reduction unit 506 is operable to receive extracted data elements with a dimension of 100×k and reduce the dimension of the extracted data elements 346 to 100×1 through logical operations, arithmetic operations, or a combination of both to produce sparsity-reduced elements 548. Sparsity reduction unit 506 is configured to output sparsity-reduced elements 548 to cascade unit 508.

[0097] Cascading unit 508 is configured to rearrange and cascade sparse reduction elements 548 to generate cascaded elements 550. For example, processor 104n may be configured to access data elements, which include inputs to a neural network layer and a corresponding set of weight values ​​(e.g., sparse elements No.1 to No.100 of database table No.1). Before returning the retrieved sparse element No.5 (e.g., a second weight value), the operable processor 104n returns the retrieved sparse element No.10 (e.g., a first weight value) to sparse reduction unit 506. Cascading unit 508 is configured to rearrange the earlier received sparse elements.The later received sparse element No. 5, which is ordered before No. 10, is configured to concatenate sparse elements No. 1 through No. 100 as concatenated element 550.

[0098] The compression / decompression unit 510 is configured to compress the concatenated element 550 to generate a dense matrix 552 of the node network 520. For example, the compression / decompression unit 510 may compress zero values ​​in the concatenated element 550 to improve the bandwidth of the node network 520. In some cases, multiple zero values ​​may be generated in response to an arithmetic operation between the input and the weights, such as a multiplication between an input with zero values ​​and a weight value, which produces a matrix of concatenated elements 550 with multiple zeros as the result of the multiplication. The compression / decompression unit 510 may compress the matrix to generate an exemplary dense matrix 552 of the node network 520. Specification 13 / 17 pages 15 CN 121168543 A

[0099] In some embodiments, the compression / decompression unit 510 can decompress the received dense matrix. For example, a first processor 104n (e.g., processor 9 in FIG. 2) can receive a dense matrix from an adjacent second processor 104n (e.g., processor 10 in FIG. 2) via a node network 520. The adjacent second processor 104n can decompress the received dense matrix and can concatenate the decompressed dense matrix with concatenation elements 550 to form an updated concatenation element that can be compressed, and then output it to the node network 520.

[0100] FIG. 6 is a flowchart illustrating an exemplary process 600 for generating the output of a neural network layer. The process 600 can be implemented or performed using the system 100 described above. The description of the process 600 can be referenced to the computing resources of the system 100 described above. In some embodiments, the steps or actions of the process 600 are enabled by programming firmware or software instructions that can be executed by one or more processors and resources of the device described in this document.

[0101] Referring now to process 600, the host of system 100 receives batch inputs to a neural network layer (602). For example, host 102 may receive one or more batches of input for processing by multi-core processing unit 104. Each input in one or more batches is stored in a memory location identified by an address. For at least one batch of input, circuitry of system 100 identifies one or more duplicate addresses in the address list for each input in the batch (604). For example, scattering circuitry 202 is operable to scan the address list to identify one or more duplicate addresses in the address list for each input in the batch.

[0102] For each identified duplicate address, circuitry generates a unique identifier (606) to identify the duplicate address in the address list. For example, scattering circuitry 202 is operable to generate a corresponding unique identifier for each duplicate address in the address list.The system identifies specific addresses as duplicates. The address list can be used to form a batch of input features, which include multiple sets of input features. In the corresponding set of input features of the input feature samples, the circuitry of the system 100 can generate a unique identifier for each duplicate address.

[0103] Based on one or more unique identifiers generated by the circuitry, the system 100 generates a filtered address list (608) of the batch of inputs. For example, the host 102 interacts with the distributed circuitry 202 to receive a corresponding unique identifier for each duplicate address and generates the filtered address list of the batch of inputs based at least on one or more unique identifiers generated by the distributed circuitry 202.

[0104] For this batch of inputs, the system 100 obtains a first input (610) from a memory location identified by an address corresponding to a unique identifier based on the filtered address list. For example, based on the filtered address list, the system 100 can use a specific processor assigned to a corresponding data slice to obtain the first input from a memory location of the data slice, the memory location being identified by an address corresponding to a unique identifier.

[0105] To obtain a first input, the request processor may provide a request to a remote processor for processing by the remote processor to obtain the first input. The request may include the address of the first input corresponding to a unique identifier. In some cases, the request and the address may be sent separately, but simultaneously or in parallel. In some cases, the request and the address may be sent sequentially, with one before the other.

[0106] In response to the remote processor processing the request, the request processor receives the first input from the remote processor, the first input being stored at a memory location of a data slice allocated to the remote processor. In some embodiments, the request processor may be operable to perform one or more reduction operations to generate the output of a neural network layer from the first input, as described in more detail below, the first input being obtained from the data slice.

[0107] System 100 may determine partitions of addresses in a filtered list between each of the plurality of processors in multi-core processing unit 104. For example, system 100 may determine that the data element associated with address

[26] is partitioned into a data slice allocated to processor 13, the data elements associated with addresses

[96] and

[99] are partitioned into a data slice allocated to processor 15, or the data element associated with address

[912] is partitioned into a data slice allocated to processor 3. Specification 14 / 17 pages 16 CN 121168543 A

[0108] In this way, and for each partition of the address, the system can obtain a first input from the memory location identified by the address in the address partition using the corresponding processor and other corresponding data elements (e.g., weight values) in the data slice, the corresponding processor is assigned to retrieve the input, and the data slice is assigned to the corresponding processor. For example, the address can beEach partition is assigned to a data slice having activation or gain values ​​corresponding to vector elements of a multidimensional sparse tensor. In some embodiments, each data slice with activation or gain values ​​is assigned to a specific processor for multiplying with a corresponding weight vector to perform neural network computation.

[0109] System 100 generates the output of a neural network layer, such as an embedding output (612), from the obtained first input. In some embodiments, the neural network layer is an embedding layer of a neural network, and the embedding output includes an embedding feature vector. Distributed circuitry 202 is configured to determine a reverse mapping between a unique identifier and a duplicate address of a specific input in a batch of inputs to the embedding layer. In some embodiments, a reverse mapping between a unique identifier and a duplicate address is determined for a specific input with multiple sets of input features.

[0110] Based on the reverse mapping from a unique identifier to a duplicate address of a specific input, system 100 can be operated to generate one or more vectors (e.g., embedding feature vectors) as the output of the embedding layer. For example, generating the output of a neural network layer (e.g., embedding output) based on the reverse mapping may include analyzing a list of addresses parsed for duplicate addresses to map the embedding feature vector back to a specific input in a batch of inputs to the neural network layer, the embedding feature vector being generated for the duplicate address and corresponding to the unique identifier.

[0111] In some examples, generating the output of the embedding layer may include generating one or more dense matrices in response to processed data associated with a sparse table of embedded values. The data associated with the table may be represented in matrix form. The data may be manipulated using the processor and computing resources of system 100 based on linear algebra algorithms including matrix multiplication. The matrix may be a one-dimensional vector or a multi-dimensional matrix. The matrix may be represented by a data structure, such as a database table or variables. The embedded table may be fragmented in, for example, row and column dimensions and stored in a portion of memory that is mapped to a specific processor among the plurality of processors forming multi-core processing unit 104.

[0112] The subject matter and functional operations described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures and structural equivalents disclosed herein, or in combinations of one or more of them. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus.

[0113] Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to an appropriate receiving device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, or a random or serial storage medium.Access memory devices, or combinations thereof.

[0114] The term “computing system” includes various means, devices, and machines for processing data, including, for example, programmable processors, computers, or multiprocessors or multiple computers. A means may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, means may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0115] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, or as a component, subroutine, or other unit suitable for a computing environment.

[0116] A computer program may correspond to, but is not necessarily required to correspond to, a file in a file system. The program may be stored in a portion of a file that holds other programs or data, for example, in a markup language document, in a single file dedicated to the program, or in one or more scripts in multiple coordination files, such as files that store one or more modules, subroutines, or code sections. The computer program may be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0117] The processes and logical flows described in this specification may be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows may also be executed by dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units), and the apparatus may also be implemented as dedicated logic circuitry.

[0118] For example, a computer suitable for executing a computer program may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory, random access memory, or both. Some components of a computer are a central processing unit for executing and implementing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices.The aforementioned high-capacity storage devices are used to store data, such as disks, magneto-optical disks, or optical disks. However, a computer does not necessarily have to have such a device. Furthermore, to name just a few examples, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive.

[0119] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CDROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0120] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as an LCD (liquid crystal display) monitor, to display information to the user, and the computer has a keyboard and pointing device, such as a mouse or trackball, to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0121] Embodiments of the subject matter described herein may be implemented in a computing system that includes back-end components, such as a data server, or middleware components, such as an application server, or front-end components, such as a client computer with a graphical user interface or a web browser, through which the user may interact with an implementation of the systems and technologies described herein, or any combination of these back-end, middleware, or front-end components. The components of the system may be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0122] The computing system may include clients and servers. Clients and servers are typically located far apart and usually interact via a communication network. The client-server relationship arises from computer programs running on their respective computers, as described on pages 16 / 17 of the specification (CN 121168543 A), and there is a client-server relationship between them.

[0123] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0124] Similarly, although operations are described in a specific order in the drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0125] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions in the claims may be performed in a different order and still yield the desired result. As an example, the processes shown in the figures do not necessarily require the specific order or sequence shown to obtain the desired result. In some embodiments, multitasking and parallel processing may be advantageous. Instruction manual, page 17 / 17, 19 CN 121168543 A, Figure 1; Instruction manual, Figure 1 / 9, page 20 CN 121168543 A, Figure 2; Instruction manual, Figure 2 / 9, page 21 CN 121168543 A, Figure 3; Instruction manual, Figure 3 / 9, page 22 CN 121168543 A, Figure 4A; Instruction manual, Figure 4 / 9, page 23 CN 121168543 A, Figure 4B; Instruction manual, Figure 5 / 9, page 24 CN 121168543 A, Figure 4C; Instruction manual, Figure 6 / 9, page 25 CN 121168543 A, Figure 4D; Instruction manual, Figure 7 / 9, page 26 CN 121168543 A, Figure 5; Instruction manual, Figure 8 / 9, page 27 CN 121168543 A, Figure 6; Instruction manual, Figure 9 / 9, page 28 CN 121168543 A, Translation of Abstract inEnglish Abstract Methods, systems, and apparatus, including computer-readable media, are described for performing neural network computations using a system configured to implement a neural network on a hardware circuit. The system includes a host that receives a batch of inputs to a neural network layer. Each of the inputs is stored in a memory location identified by an address. The system identifies one or more duplicate addresses in a listing of addresses for one or more inputs. For each duplicate address: the system generates a unique identifier that identifies the duplicate address in the listing of addresses. The system (i) obtains first inputs from memory locations identified by addresses corresponding to the unique identifiers and (ii) generates an output of the layer from the obtained first inputs. (CN 121168543A)

Claims

1. A method of performing neural network computation, the method comprising: comprises: in a request from a first processor, identifying a set of addresses of locations in a memory; the first processor performing a deduplication operation on the set of addresses by: identifying duplicate addresses in the set of addresses; generating unique identifiers that map to the duplicate addresses; and filtering the duplicate addresses from the set according to the unique identifiers; after filtering the duplicate addresses from the set, a second processor retrieving data from memory locations identified by addresses in the set of addresses using the unique identifiers; and generating an output in response to processing the data using the second processor.

2. The method of claim 1, wherein: the data comprises i) a set of input features and ii) a vector of values representing trainable weights of a neural network layer; and the unique identifiers are for duplicate addresses of memory locations storing feature values of input features in the set. identifying duplicate addresses in the set of addresses comprises:

3. The method of claim 2, wherein, identifying two or more input features in the set of input features having the same feature value. the vector of values is an embedding of an embedding table stored on a plurality of data shards of the memory.

4. The method of claim 2, wherein, the neural network layer is an embedding layer of an artificial neural network, and processing the data using the second processor comprises:

5. The method of claim 4, wherein, providing the feature values of the data as input to the embedding layer; and processing the data at the embedding layer of the neural network. generating the output comprises generating one or more embedding feature vectors as output from the embedding layer.

6. The method of claim 5, wherein, generating the unique identifiers comprises:

7. The method of claim 6, wherein, generating a hash ID based on a memory offset of one of the duplicate addresses; and generating the unique identifier based on the hash ID. performing the method using an integrated circuit comprising a set of distributed processors, the set of distributed processors comprising the first processor and the second processor.

8. The method of claim 1, wherein, performing the deduplication operation comprises:

9. The method of claim 8, wherein, filtering out addresses that are duplicated in the set of addresses to mitigate load imbalance between the first processor and the second processor. performing the deduplication operation comprises:

10. The method of claim 8, wherein, filtering out addresses that are duplicated in the set of addresses to reduce overall computational load of at least the second processor in the set of distributed processors. a non-transitory machine-readable medium storing instructions executable by a processing device to perform the method of any of claims 1-10.

11. A non-transitory machine-readable storage device, comprising: comprises:

12. A system to perform neural network computations, the system comprising: a plurality of processors; and a non-transitory machine-readable medium storing instructions executable by one or more of the plurality of processors to perform the method of any of claims 1-10. ​ ​

Images