On-chip interconnect for memory channel controllers

Abstract

An integrated circuit method, system, and apparatus (including computer-readable media) for accelerating machine learning computations are described. The circuit includes a plurality of processor cores, each processor core including a plurality of channel controllers, an interface controller for connecting each channel controller to any memory channel of a system memory, and a fetch unit in each channel controller. Each fetch unit is configured to receive channel data encoding addressing information, retrieve data from any memory channel of the system memory using the interface controller based on the addressing information, and write the retrieved data to a vector memory of the processor core via each channel controller including a corresponding fetch unit.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 167593, filed March 29, 2021, the entirety of which is incorporated herein by reference.

[0002] This application is related to U.S. Provisional Application No. 63 / 001216, filed March 27, 2020, the entirety of which is incorporated herein by reference.

[0003] This application is related to U.S. Application No. 16 / 865,539, filed May 4, 2020, the entirety of which is incorporated herein by reference. [Background technology]

[0004] background This specification relates generally to using circuits to perform neural network computations.

[0005] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input for one or more other layers of the network, e.g., other hidden layers or other output layers of the network. Some layers of the network generate outputs from the received input according to the current values ​​of each set of parameters.

[0006] Some neural networks are convolutional neural networks (CNNs) (e.g., for performing image processing) or recurrent neural networks (RNNs) (e.g., for performing speech and language processing). Each of these neural networks includes a corresponding set of convolutional neural network layers or recurrent neural network layers. Each neural network layer may have an associated set of kernels and an embedding layer for processing the inputs to generate a set of vectors for training the neural network. The kernels may be represented as weighted tensors, i.e., multidimensional arrays. As an example, the embedding layer may process a set of inputs, such as inputs of image pixel data or activation values ​​generated by the neural network layer. The set of inputs or the set of activation values ​​may also be represented as a tensor. Summary of the Invention [Problem to be solved by the invention]

[0007] overview The distributed system may include a memory for storing values ​​that are accessed and used to perform operations or compute values. Each value may be stored at a location in the memory identified by an address. The memory may be configured to include different memory channels. Each channel includes a set of memory locations identified by a corresponding set of addresses. A channel controller is used to retrieve data specified by a request by controlling and managing access to specific memory locations in a given memory channel. More specifically, the channel controller manages the flow of data to and from the memory using the communication channels of the distributed system.

[0008] Based on this context, a technique for implementing a hardware interface controller is described. The hardware interface controller is a shared interconnect or crossbar device configured to enable any channel controller of an integrated circuit to communicate with any memory channel of an exemplary system memory. For example, the interface controller is configured to provide dynamic connections between a given channel controller and various memory channels of a high-bandwidth memory. The channel controller may be included in the hardware architecture of the interface controller. The integrated circuit may be a hardware machine learning accelerator or other dedicated processor including multiple processor cores. The interface controller may be integrated into the processing pipeline of the accelerator's circuitry to enable each channel controller to read data from and write data to any channel of the high-bandwidth memory system.

[0009] An aspect of the subject matter described herein may be embodied as an integrated circuit configured to accelerate machine learning computations. The circuit includes a plurality of processor cores, each processor core including a plurality of channel controllers, an on-chip interconnect (OCI) interface configured to connect each channel controller of the plurality of channel controllers to a respective memory channel of a system memory, and a respective fetch unit in each channel controller of the plurality of channel controllers. Each fetch unit is configured to: i) receive a request encoding addressing information; ii) retrieve data from any memory channel of the system memory using the interface controller based on the addressing information; and iii) write the retrieved data from any memory channel to a vector memory of the processor core via the channel controller including the fetch unit.

[0010] Optionally, these and other implementations may each include one or more of the following features. For example, in some implementations, the interface controller includes or is operable to control an in-memory OCI node that includes: i) a number of read interfaces used to retrieve data from any memory location along any memory channel of the system memory; and ii) a number of write interfaces used to write data to any memory location along any memory channel of the system memory.

[0011] In some implementations, the in-memory OCI node includes a first-in-first-out (FIFO) memory queue configured to store incoming OCI transaction requests during arbitration of an existing OCI transaction to i) retrieve data from any memory location in the system memory or ii) write data to any memory location in the system memory. In some implementations, the integrated circuit further includes an interface controller configured to receive a request encoding addressing information, generate a plurality of control signals based on a plurality of IDs in the addressing information, and provide the plurality of control signals to any memory channel in the system memory to fetch data stored in the memory location in the system memory.

[0012] In some implementations, the addressing information is derived from one or more incoming OCI transaction requests received by the interface controller, where at least one existing OCI transaction request for accessing any memory location in the system memory is pre-stored in a FIFO memory queue. The addressing information can be derived from one or more incoming OCI transaction requests, where each incoming OCI transaction request can include multiple identifiers (IDs) corresponding to any set of memory locations in any memory channel in the system memory.

[0013] In some implementations, the OCI interface includes two or more addressing modes, and the interface controller is operable to generate control signals to enable selection between the two or more addressing modes of the OCI interface. A first addressing mode of the two or more addressing modes is a channel interleave mode that constrains mapping of the embedded tables to memory channels of the system memory. A second addressing mode of the two or more addressing modes is a stack interleave mode that extends mapping of the embedded tables to memory channels of the system memory.

[0014] In some implementations, each OCI transaction request is received by a fetch unit and processed by an interface controller to initiate access to any memory location indicated in the OCI transaction request. In some implementations, each OCI transaction request to access data stored in system memory encodes i) a 32-byte length corresponding to the data being accessed, ii) a token ID representing a unique source address of any memory location in the system memory, and iii) a destination address of a memory location in the vector memory. In some implementations, the embedded table is partitioned as multiple data shards that are allocated across memory channels across the system memory, and the interface controller selects a stack interleaving mode to access any portion of the embedded table with any of the multiple channel controllers.

[0015] The integrated circuit further includes a respective flash unit in each channel controller of the plurality of channel controllers, each flash unit configured to: i) receive a request encoding addressing information; and ii) write data from a source address in the vector memory to any memory location of the system memory using the interface controller based on the addressing information. In some implementations, machine learning computations are performed to process neural network inputs through neural network layers, and the flash unit is used to write parameters of the neural network layers to any memory location of the system memory using the interface controller.

[0016] Another aspect of the subject matter described herein may be embodied as a computer-implemented method performed with an integrated circuit configured to accelerate machine learning computations. The integrated circuit comprises a system memory and a processor core including a plurality of channel controllers. The method includes receiving a plurality of requests, each of the plurality of requests encoding addressing information. For a first request of the plurality of requests, the method includes identifying an arbitrary memory location of an arbitrary memory channel of the system memory based on the addressing information of the first request, obtaining data from a memory location in the arbitrary memory location of the system memory using an on-chip interconnect (OCI) interface for connecting each channel controller of the plurality of channel controllers to each memory channel of the system memory, and writing the obtained data from the memory location to a vector memory of the processor core based on the addressing information of the first request.

[0017] Optionally, these and other implementations may each include one or more of the following features: For example, in some implementations, the OCI interface includes an in-memory OCI node including multiple read interfaces and multiple write interfaces, and the method further includes retrieving data from any memory location along any memory channel of the system memory using the multiple read interfaces based on control signals generated by the interface controller, and writing data to any memory location along any memory channel of the system memory using the multiple write interfaces based on control signals generated by the interface controller.

[0018] In some implementations, the in-memory OCI node includes a first-in-first-out (FIFO) memory queue, and the method includes storing incoming OCI transaction requests in the FIFO memory queue during arbitration of an existing OCI transaction that i) retrieves data from any memory location in the system memory or ii) writes data to any memory location in the system memory.

[0019] In some implementations, the method further includes an interface controller receiving a request to encode addressing information, the interface controller generating a plurality of control signals based on a plurality of IDs in the addressing information, and the interface controller providing the plurality of control signals to any memory channel of the system memory to fetch data stored in a memory location of the system memory.

[0020] In some implementations, the addressing information is derived from one or more incoming OCI transaction requests received by the interface controller, where at least one existing OCI transaction request for accessing any memory location in the system memory is pre-stored in a FIFO memory queue. In some implementations, the addressing information is derived from one or more incoming OCI transaction requests, where each incoming OCI transaction request includes a number of identifiers (IDs) corresponding to any set of memory locations in any memory channel in the system memory.

[0021] Other implementations of this and other aspects include systems, devices, and computer programs configured to perform the operations of the methods encoded on a computer storage device. One or more computer systems may be configured with software, firmware, hardware, or combinations thereof that are installed on the system and that, when operated, cause the system to perform the operations. One or more computer programs may be configured to include instructions that, when executed by a data processing device, cause the device to perform the operations.

[0022] The subject matter described herein may be implemented in particular embodiments to realize one or more of the following advantages.

[0023] The circuitry for the crossbar / on-chip interconnect may be implemented in a dedicated hardware circuit, such as a hardware accelerator used for a distributed system. The crossbar allows each channel controller to read data from and write data to any address location of a memory cell in any channel of a high-bandwidth memory system that communicates with a processor core or accelerator chip. This eliminates the need to map a channel controller to a specific memory channel, avoiding performance degradation due to load imbalance. The crossbar is implemented to distribute the address allocation load by assigning addresses to be processed by all memory channels to any channel controller. Thus, the crossbar can improve the performance of the distributed system compared to conventional methods.

[0024] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the detailed description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary computing system. [Diagram 2] FIG. 2 is a block diagram illustrating an architecture including an example control unit, channel controller, and memory channel. [Diagram 3] FIG. 2 illustrates an exemplary algorithm used to perform memory channel controller load balancing. [Figure 4] FIG. 2 illustrates an exemplary allocation of requests to different channel controllers. [Figure 5A] FIG. 2 illustrates an example OCI interface controller. [Figure 5B] FIG. 2 illustrates an example OCI interface controller. [Figure 6]FIG. 13 is an exemplary flow diagram illustrating a write operation involving an OCI processor node. [Figure 7] FIG. 13 is an exemplary flow diagram illustrating a read operation involving an OCI processor node. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description 1 is a block diagram illustrating an example computing system 100 configured to retrieve data elements stored in a memory of the system 100. The data elements may be retrieved and used to perform neural network computations of an example machine learning workload. For example, the data elements may be processed to perform embedding layer operations to compute outputs of a neural network layer or to generate an embedding set for training a neural network.

[0027] The embedding outputs are generated when training the neural network of the system 100 to perform a particular computational function, such as computations related to machine translation, natural language understanding, ranking models, or content recommendation models. In some implementations, training the neural network includes updating a pre-stored embedding set in an embedding table of the neural network, e.g., during a prior stage of training the neural network. For example, embeddings of an embedding layer of a neural network may be trained together with the neural network that uses the embeddings. Thus, using the techniques described herein, embeddings can be updated during training of a neural network, improving efficiency over conventional approaches.

[0028] Typically, an embedding layer of a neural network is used to embed features into a feature / embedding space corresponding to the embedding layer. An embedding vector may be a vector of numerical values ​​that are mapped to corresponding features in a feature set of a lookup table representing the embedding layer. A feature may be an attribute or characteristic shared by independent units that perform analysis or prediction. For example, the independent units may be groups of words in a vocabulary, or image pixels that form part of items such as images and other documents. An algorithm for training the embeddings of the embedding layer may be executed by a neural network processor that maps features to embedding vectors. In some implementations, the embeddings of the embedding table are jointly trained with other layers of the neural network that use embeddings. Such training is done by backpropagating gradients to update the embedding table.

[0029] In other implementations, the embeddings may be learned separately from other layers of the neural network that use them, for example when the embeddings are pre-trained. For example, a neural network processor may compute the embeddings by processing information about the discrete input features with an algorithm to determine a mapping or arrangement of similar inputs to geometrically close embedding vectors in the embedding space. In some cases, the process of computing embeddings may represent a feature learning or feature engineering technique that allows the system to automatically discover the representations needed to detect features from raw input data.

[0030] In some implementations, a given "input" can have one or more features of one or more types, and the embedding layer generates an embedding corresponding to each type. For example, the input can be a search query input having several different feature types. These feature types can include characteristics of a user or a user device (e.g., location, preferences, device type), query tokens, previously submitted queries, or other related types that may correspond to attributes of the search query. For any feature type having multiple features for a given input, the computing system is operable to obtain a respective embedding for each feature. The system is also operable to generate a final embedding for that feature type, for example, by combining the obtained embeddings and calculating an average of the embedding values.

[0031] The computing system 100 includes a host 102, a multi-core processing unit 104, and a memory unit 105 ("memory 105"). The memory 105 includes data shards 106a-106k (where k is an integer greater than 1). The memory 105 is described in more detail below. In general, the host 102 may be a processing unit, such as, for example, a processor, multiple processors, or multiple processor cores. Thus, the host 102 may include one or more processors and is operable to generate or process instructions for accessing a target dense matrix and generate the target dense matrix by transmitting instructions 110 to the multi-core processing unit 104. As described in more detail below, performing the embedding layer operation may include generating a dense matrix by transforming sparse elements from one or more matrices.

[0032] The multi-core processing unit 104 accesses corresponding elements 108a-108n (n is an integer greater than 1) from one or more of the data shards 106a-106k in the memory 105. The multi-core processing unit 104 generates a target dense matrix 112 using the corresponding elements 108a-108n and provides the target dense matrix 112 to the host 102 for further processing. The multi-core processing unit 104 may generate the target dense matrix 112 by converting each element of the elements 108a-108n into a vector and concatenating the n vectors into a single vector.

[0033] In general, in the context of embedding, the "sparse" information corresponding to the sparse elements may be a one-hot vector that identifies feature values. For example, if a given feature has five possible values ​​(e.g., A, B, C, D, E), the sparse vector identifies feature value "A" as (1,0,0,0,0), and the embedding layer maps (1,0,0,0,0) to a dense embedding vector for feature value "A." In some implementations, during training of the embedding layer to learn embeddings, elements 108a-108n may be weight values ​​of the embedding table that are converted to vectors, such as an embedding vector for feature value "B" or "C." The weight values ​​may be converted using a neural network processor of multi-core processing unit 104 that executes a training algorithm to compute embeddings based at least on the mapping of features to embedding vectors.

[0034] The host 102 can process instructions to update the target dense matrix and send the updated dense matrix to the multi-core processing unit 104. For example, the target dense matrix may correspond to an embedding of a neural network. Thus, the host 102 can process instructions to update the embedding to generate an updated dense matrix. For example, during a subsequent iteration of training the neural network to update the embedding, a backward pass can be performed to update the embedding by determining a new mapping between the input features and the embedding vectors and generating an updated dense matrix based on the new mapping. In some implementations, the multi-core processing unit 104 is operable to convert the updated dense matrix into a corresponding sparse element and update one or more sparse elements (e.g., weights) stored in the data shards 106a-106k accordingly.

[0035] As described above, the host 102 is configured to process instructions for execution within the computing system 100. In some implementations, the host 102 is configured to process the target dense matrix 112 generated by the multi-core processing unit 104. In some other implementations, the host 102 may be configured to request the multi-core processing unit 104 to generate the target dense matrix 112, and another processing unit may be configured to process the target dense matrix 112.

[0036] Each processor of the multi-core processing unit 104 is configured to retrieve data elements stored in a memory of the system 100. The memory may include a number of data shards 106a-106k that store data including elements 108a-108n. The data may include inputs, activations, gain values, or weight values ​​corresponding to parameters or kernels of a weight matrix structure. In some implementations, the data shards 106a-106k may be volatile memory units. In some other implementations, the data shards 106a-106k may be non-volatile memory units.

[0037] A data shard, as used herein, may include a memory bank, a memory cell, or an associated section of memory, such as on-chip memory, high bandwidth system memory, external memory, or a combination thereof. The data shards 106a-106k may also be another form of computer-readable medium, such as a storage area network or other configuration of devices. The data shards 106a-106k may be connected to the multi-core processing unit 104 using electrical, optical, or wireless connections. In some implementations, the data shards 106a-106k may be part of the multi-core processing unit 104 and rely on a processor-in-memory (PIM) architecture.

[0038] The multi-core processing unit 104 is configured to determine a dense matrix based on the sparse elements. The multi-core processing unit 104 includes multiple interconnected processors or processor cores. For example, the multi-core processing unit 104 may be a distributed processing system including multiple interconnected processor cores. In general, the terms "processor" and "processor core" may be used interchangeably to describe the interconnected individual processing resources of the multi-core processing unit 104.

[0039] The system 100 further includes a process ID control unit 114 ("control unit 114"). The control unit 114 receives the set of ID headers and performs operations to distribute the ID headers or distribute portions of the information contained in the ID headers. As described in more detail below with reference to FIG. 2, the ID headers are distributed to a channel controller. In some implementations, the system 100 includes multiple control units 114. For example, the system 100 may include a control unit 114 for each processor or processor core of the system 100. Each control unit 114 connected to a processor / core of the multi-core processing unit 104 receives the set of ID headers from a source. The source may be the host 102 or another processor of the multi-core processing unit 104.

[0040] The ID header may represent a request that includes information that addresses a memory location in memory 105. Memory 105 may represent a high bandwidth memory (HBM) or input / output (I / O) device that exchanges data communications with control unit 114 in a processor core of an exemplary hardware circuit included in system 100. For example, memory 105 may exchange data communications with a processor core of multi-core processing unit 104 to send input to the processor core and receive output generated by one or more computing resources of the processor core. The inputs and data values ​​stored or written to memory locations of memory 105 may represent arrays of vector elements or vector values.

[0041] Memory 105 may be a dynamic random access memory (DRAM) asset of system 100. In some implementations, memory 105 is external or off-chip memory relative to an exemplary hardware circuit including one or more processors or processor cores. Memory 105 is configured to exchange data communications with on-chip resources of the hardware circuit, such as a vector processing unit (VPU) (described below) or vector memory banks of the VPU. For example, memory 105 may be located at a physical location external to an integrated circuit die representing the hardware circuit of system 100. Thus, memory 105 may be remote or non-local to computing resources located within the integrated circuit die. Alternatively, memory 105 or some of its resources may be located within an integrated circuit die representing a dedicated hardware circuit. As a result, memory 105 is local to or co-located with the computing resources of the circuit.

[0042] 2 is a block diagram illustrating an architecture 200 including an example channel controller 202, memory channels 204 of memory 105, and the control unit 114 described above. Each of the memory channels 204 may represent a memory bank of memory 105, a set of memory banks of memory 105, a set of memory locations of memory 105, or a combination thereof.

[0043] The set of channel controllers 202 includes at least a number of channel controllers, denoted as C0, Cl, C2, and C15, respectively. In the example of FIG. 2, the architecture 200 may include 16 channel controllers. In some implementations, the architecture 200 includes more or fewer channel controllers. For example, the architecture 200 may include N channel controllers and N memory channels 204. These aspects of the architecture 200 are indicated by reference numerals 202-n, which indicate individual channel controllers, and 204-n, which indicate individual memory channels.

[0044] 2 illustrates an example in which individual channel controllers, such as channel controllers 202-0 (C0) and 202-2 (C2), are hard-mapped to specific memory channels, such as corresponding memory channels 204-0 (C0) and 204-2 (C2), respectively. As described above, system memory 105 having channel controllers 202 that are hard-mapped to specific memory channels may experience load imbalances. These load imbalances may stall or significantly delay operation of neural network computations taking place in system 100, such as operations to generate outputs for embedded layers.

[0045] This conventional approach of mapping a particular channel controller 202 to a particular memory channel may have other challenges. For example, this approach has the constraint of requiring data to be stored in a manner that is sensitive to mapping addresses and data to a particular channel controller 202. Furthermore, this approach is inefficient when the system needs to retrieve a vector from a large space in memory by performing a large number of randomized lookups. To address these challenges, an on-chip interconnect (OCI) or crossbar (described below) is integrated into the dedicated hardware circuit. By integrating the crossbar into the processing pipeline of the accelerator circuit, each channel controller can read data from and write data to any channel of the high-bandwidth memory system.

[0046] In some implementations, the dedicated circuitry is a multi-core hardware accelerator and the OCI is a channel controller interface that is uniquely configured based at least on the multi-core structure of the hardware accelerator. For example, the channel controller interface is configured to enable communication between each core of the multi-core hardware accelerator and each memory channel of memory 105 that includes a different type of memory structure corresponding to the memory channel.

[0047] The channel controller interface may be configured to be sized 32B×4 instead of 128B×1. Based on this exemplary size configuration, the channel controller interface may include multiple independent transaction threads between the memory 105 and the channel controller 202 without the need for extra external ports for OCI hardware. In some implementations, the channel controller interface is configured to efficiently handle the dynamic bandwidth requirements of each channel for different computation phases. For example, the gigabytes per second (GBps) bandwidth requirements may change in response to different computations for different access sizes, e.g., 32-byte (32B) accesses, 64-byte (64B) accesses, 128-byte (128B) accesses. The phases may include a forward pass computation, a backward pass computation, and a backward pass computation that implements an optimization algorithm, such as Adagrad, to update a learning value of a particular vector based on gradients generated by evaluating a neural network with some training data.

[0048] The channel controller interface may be uniquely configured to include multiple node interfaces. For example, the crossbar may include: i) an intra-client node interface operable to carry direct memory access (DMA) descriptors and control messages; ii) an intra-memory node interface operable to carry read / write commands and data for various memory structures (e.g., buffer memory, instruction memory, shared memory, vector memory, host memory) of the memory system; iii) an intra-processor node interface (lower) operable to carry load / store traffic from a first / lower set of channel controllers 202 to memory 105; and iv) an intra-processor node interface (upper) operable to carry load / store traffic from a second / upper set of channel controllers 202 to memory 105. This is described in more detail below with reference to at least Figures 5A and 5B.

[0049] The channel controller interface allows a set of channel controllers to access any memory channel / address of the memory 105. However, even if the addresses specified in the request are spread among the set of channel controllers 202, a large-scale execution of a particular machine learning workload may exhibit data access patterns that result in a particular channel controller receiving a majority of the data processing load relative to the other channel controllers. In the example of FIG. 2, the channel controller 202-0 exhibits an imbalance in which its channel receives a majority of the data processing load relative to the other channel controllers (e.g., C1, C2). To address this issue, a crossbar is used to implement a particular control scheme for controlling the allocation of addresses or requests to each channel controller 202. This control scheme causes addresses to be allocated substantially evenly among the channel controllers 202. This is described in more detail below with reference to FIG. 3.

[0050] FIG. 3 illustrates an example algorithm 300 that may be used to perform load balancing of memory channel controllers 202.

[0051] As discussed above, data accesses of an exemplary machine learning workload may exhibit certain anomalous patterns. For example, even if a set of requests and addresses are generally spread across the channel controllers 202, there may be certain patterns where a particular channel controller needs to operate on a significant number of large features or large vectors. Such patterns may cause the control unit 114 to distribute a set of processing tasks or ID headers, still resulting in load imbalance of the channel controllers 202. For example, these patterns may have a bursty nature that appears over a short processing time window, such as 20-100 cycles. This load imbalance may occur even if any one of the channel controllers 202 is configured to access any memory location of the memory 105 and any memory channel 204.

[0052] Algorithm 300 corresponds to the control scheme described above and is an exemplary distribution algorithm used to perform load balancing of memory channel controllers 202 of system 100. Algorithm 300 may include pseudocode shown in the example of FIG. 3, which represents one or more instruction steps of distribution algorithm 300. In some implementations, algorithm 300 is a modified round robin distribution algorithm. The modified round robin attribute of distribution algorithm 300 allows for parsing and distributing a set of ID headers to channel controllers 202.

[0053] For example, the modified round robin distribution algorithm 300 is configured to interrupt or suppress potential anomalous sequences that may occur during data access of a machine learning workload. To this end, the modified round robin distribution algorithm 300 is configured to enable allocation of ID headers (e.g., activation or gradient addresses) in a manner that distributes the load across each channel controller 202 in the set of channel controllers 350. A standard round robin approach to scheduling processes indicates the selection of channel controllers in a simple circular order, without priority.

[0054] To address the bursty patterns discussed above, the round robin approach may be modified or altered to first detect an initial completion of a first cyclic selection round. In response to detecting the initial completion, the control unit 114 may adjust the increment parameter to modify the initial channel controller selected for a second or subsequent cyclic selection round.

[0055] For example, the system 100 may include 16 channel controllers (e.g., CC0-CC15). The control unit 114 may select each channel controller 202 during an initial round and detect the completion of the initial round based on a count parameter indicating that CC15 was selected during the initial round. The count parameter may correspond to the total number of channel controllers (16). Thus, selection of CC15 during the initial round indicates selection of each of the 16 channel controllers. The control unit 114 may then adjust the value of the increment parameter to bypass selection of certain channel controllers.

[0056] For example, the control unit 114 may bypass selection of CC0 and select CC1 at the beginning of a subsequent round of channel selection by increasing the increment parameter. Similarly, the control unit 114 may bypass selection of CC1 and select CC2 at the beginning of another subsequent round of channel selection by again increasing the increment parameter. In some implementations, as described in more detail below with reference to FIG. 4, the control unit 114 may increase (or decrease) the channel count increment by periodically adjusting the value of the increment parameter based on one or more observed data access patterns.

[0057] FIG. 4 shows a table 400 illustrating an example sequence 410 for selecting a channel controller 202 to balance the allocation of requests to different channel controllers 202 .

[0058] As briefly described above, a native round robin scheme may suffer from unusual patterns in the input data being accessed for computation. For example, there may be a pattern where every 16th ID header belongs to an embedding table with the longest embedding vector and the most computationally intensive optimizer. This exemplary pattern may cause load imbalance in a native round robin scheme. The control unit 114 may be a hardware component of a processor core that executes instructions corresponding to the distribution algorithm 300 that implements a modified round robin ID header distribution scheme.

[0059] The distribution scheme is operable to reduce the probability of load imbalance due to anomalous patterns in the set of input data based on the algorithm 300. Using the algorithm 300, an exemplary sequence 410 for selecting a channel controller 202 can be generated. Each number in the sequence indicates a channel controller to be selected. In some implementations, the sequence 410 can first iteratively select each channel controller in the set of channel controllers (e.g., 0-15) based on an unmodified initial round robin flow.

[0060] After the first iteration of selecting each channel controller, the round robin flow may be modified to select channel controller CC1 rather than selecting channel controller CC0 again at the beginning. Similarly, after the second iteration of selecting each channel controller, the round robin flow may be modified to select channel controller CC2 rather than selecting channel controller CC1 again at the beginning. This modified selection scheme provides an example of a method that enables the control unit 114 to select each channel controller in the set of channel controllers and distribute addresses evenly or substantially evenly among the set of channel controllers. In some implementations, the system 100 monitors the data access patterns of each channel controller and dynamically adjusts or modifies the distribution scheme based on the observed patterns.

[0061] The control unit 114 generates a set of channel numbers for a set of channel controllers 202 using the modified distribution scheme. The generated set of channel numbers is processed in the control unit 114, which forwards the ID headers to the corresponding channel controllers 204. In some implementations, the control unit 114 forwards the ID headers to the corresponding channel controllers 204 based on an exemplary sequence 410 derived from the modified distribution scheme. To ensure sufficient load balancing of the ID header processing workload across the channel controllers 202, the algorithm 300 causes the control unit 114 to implement certain characteristics for selecting channel numbers. In some implementations, the algorithm 300 is used to select channels based on the exemplary steps of the pseudocode shown in FIG. 3.

[0062] For example, the channel selection property requires that the generation of channel numbers be fair and non-bursty. A "fair" property for generating channel numbers results in (or requires) all channel controllers being selected equally or substantially equally for a given machine learning task. A "non-bursty" property for generating channel numbers results in (or requires) selecting channel controllers without intermittent increments when repeatedly selecting a particular channel controller for a given machine learning task. For example, a channel number sequence of "0, 1, 0, 1, 4, 5, 0, ..." is not a desirable pattern and does not satisfy the "non-bursty" property for generating channel numbers.

[0063] An exemplary set of metrics can be used to determine whether each of the above (e.g., fair and non-bursty) properties are met. The metrics include determining a count, an average (mean), and a median for the number of times a channel number appears for selection. For the "count" metric, the system 100 is operable to determine a count of the number of times a channel or channel number is included per processing iteration. The number should be the same for all channels 202 or channel controllers 202. If the system 100 determines that the numbers are not the same, it can detect that a particular pattern of selecting channel controllers is biased and not load balanced for a given set of operations.

[0064] For the "average value" metric, system 100 is operable to determine, for each channel number, whether the number of times the channel number appears for selection converges to N (N is an integer equal to or greater than 1) after a threshold number of iterations. For example, if system 100 includes 16 channel controllers, system 100 is operable to determine, for each channel number, whether the number of times the channel number appears for selection converges to 16 after a threshold number of iterations or ID headers. In some implementations, the threshold number of iterations varies based on the size and complexity of the data being searched and manipulated.

[0065] The "median" metric indicates the burstiness of a particular channel controller. For example, if the system 100 determines that the channel controller 204-n has a low median selection value, it receives more ID headers in bursts compared to the other channel controllers (which may indicate an imbalance). The table 400 includes example metric values ​​for each channel number for example processing iterations performed on a threshold number of 2048 ID headers. As previously mentioned, the system 100 can monitor the data access patterns of each channel controller for the metrics and characteristics described above and dynamically adjust or modify the distribution / control scheme based on the observed patterns. For example, the control unit 114 can periodically adjust the value of the increment parameter to increase (or decrease) the channel count increment based on the data access pattern.

[0066] 5A and 5B illustrate an example OCI interface 500 including a channel OCI controller 502 for managing or controlling operations related to the OCI fabric (or architecture) of the system 100. The system 100 (and the OCI interface 500) may include multiple OCI controllers 502. In some implementations, the OCI controller 502 corresponds to each channel controller 202 described above. In some other implementations, the OCI controller 502 is different from each channel controller 202 described above. The OCI controller 502 is an interface controller that includes the nodes and read or write interfaces of the OCI interface and is operable to control the OCI interface 500. In some implementations, one or more functions of the OCI interface 500 are included in and / or controllable by the interface controller.

[0067] The OCI interface 500 may be a network node of an integrated circuit that includes multiple chips. Each chip may be a processor, a processor core, or a hardware accelerator. For example, the OCI interface 500 may be composed of multiple sub-network nodes that cooperate to connect or interconnect each chip of the integrated circuit, so that data obtained from one chip can be routed to another chip through the network nodes.

[0068] In general, controller 502 controls operations associated with one or more interface devices of the OCI interface of system 100. OCI interface 500 may include a respective 32B random access load / store interface from each channel controller and may support efficient 32B read / write access of multiple threads (e.g., 8 threads) to memory 105 of each memory channel. Controller 502 is configured to update a fetch unit or flush unit of a given channel controller by reordering one or more incoming responses from memory 105, as will be described in more detail below.

[0069] In some implementations, the OCI interface includes 4×32B read / write interfaces for each channel 504, 506. These read / write interfaces can represent intra-processor OCI nodes, with each node servicing two channels 504, 506 or a channel controller corresponding to channels 504 and 506. The example of FIG. 5A shows a read operation associated with read interface 508, and the example of FIG. 5B shows a write operation associated with write interface 510. The 4×32B configuration of the OCI interface 500 allows independent transaction threads to occur between the memory 105 and the channel controllers. For example, as described above, the OCI interface 500 may be configured to be 32B×4 in size instead of 128B×1. Based on this exemplary size configuration, the OCI interface 500 can include multiple independent transaction threads between the memory 105 and the channel controllers 202 without the need for extra external ports for OCI hardware.

[0070] The OCI interface 500 is configured to issue channel credits maintained for each data interface. In some implementations, each data interface corresponds to a sub-network or node of the OCI interface 500. The controller 502 can generate one or more control signals corresponding to each channel credit. For example, the controller 502 can generate the control signals based on various IDs in the addressing information encoded in the OCI transaction request. The controller 502 can also fetch data stored in a memory location of the system memory by providing or routing the control signals to any memory channel of the system memory.

[0071] In some implementations, each node of the OCI interface 500 serves two channel controllers, and the credits on the interface node are shared between the two channel controllers. Unlike a single 128B interface that causes long transaction delays to a lightly loaded channel controller when the other channel controller is busy, in the OCI interface 500 with a 4×32B configuration, such interdependencies exist primarily (or only) between the two channels arbitrating for the same port.

[0072] The mapping between channel controllers and OCI interfaces may be fixed or dynamic. In implementations where the mapping is fixed, one port may be busy while the other is idle for a short period of time, but the overall configuration allows for distributing IDs such that all ports are equally well utilized.

[0073] The OCI interface 500 may include at least two interfaces that interact with a direct memory access (DMA) unit of the system 100, which handles bulk transfers and control messages for a given processor core. For example, the DMA unit interacts with at least two OCI interfaces: a DMA client interface and a DMA memory interface. The DMA client interface is a bulk data movement interface for BMEM, BIMEM, SMEM, VIMEM, and HMF, and includes a descriptor sub-interface for sending descriptors to the OCI node and a message sub-interface for controlling messages related to synchronization flags and interrupt operations. The DMA memory interface handles read and write requests from the OCI, and includes a read sub-interface and a write sub-interface. The read sub-interface receives read commands from the controller 502 of the OCI interface 500 and sends responses back on a 128-byte data interface. The write sub-interface receives write commands and writes data on a 128-byte interface.

[0074] BMEM is a vector memory structure for storing vectors of data and SMEM is a scalar memory structure for storing scalar data values. BIMEM is the processor core instruction memory and VIMEM is the vector instruction memory (vimem) for storing programs executed by the address handler VPU unit. In some implementations, the VIMEM across the 16 channel controllers is treated (or configured) as one large sequential memory by the DMA unit of system 100.

[0075] In some cases, the read / write interface of OCI interface 500 is part of an in-memory OCI node of system 100. This node or unit may include an input first-in-first-out (FIFO) memory queue (or buffer) that stores incoming OCI transactions while different memory blocks / stacks are being arbitrated for write or read operations. In some implementations, one or more of the incoming OCI transaction requests to be processed during an existing operation are pre-stored in the FIFO memory queue as a step in arbitrating a series of operations.

[0076] The size of this input FIFO may be determined based on the worst case BMEM access latency. In some implementations, the size of the FIFO is determined this way even though it is not the memory with the slowest BMEM access latency. This is because memories other than BMEM may only be accessed at the beginning of a batch, so any performance penalty on the OCI interface 500 as a result of these accesses is only incurred once.

[0077] The memory 105 (e.g., HBM) may include four HBM stacks interleaved with 64-byte granularity. For example, the memory 105 includes HBM stack 0 interleaved with [0-63] bytes and [128-191] bytes, HBM stack 1 interleaved with [64-127] bytes and [192-255] bytes, HBM stack 2 interleaved with [256-319] bytes and [384-447] bytes, and HBM stack 3 interleaved with [320-383] bytes and [448-511] bytes. In some implementations, the memory 105 may include more or fewer stacks and various byte granularities. With respect to 64-byte granularity, when 128 bytes are accessed from HBM stack 0, HBM stack 0 returns data stored in memory locations corresponding to [0-63] bytes and [128-191] bytes. This may represent an exemplary Q128 addressing mode on the OCI fabric.

[0078] The OCI controller 502 of the exemplary processor core can operate the processor core in a channel interleaved mode or a stack interleaved mode, each of which represents an exemplary addressing mode of the processor core. In the channel interleaved addressing mode, the embedded table is mapped to a single HBM stack closest to the processor core, so that all memory accesses are made to the closest HBM stack, which provides a shorter latency compared to another HBM stack.

[0079] In stack interleaved mode, the embedding table may be split across an on-chip high bandwidth memory (HBM) stack (e.g., memory 105) and utilize a different system addressing scheme. For example, when in stack interleaved mode, the embedding table may be accessed using the same address scheme as the rest of system 100, and software control of system 100 may be configured to use the full capacity of memory 105, even when using fewer processing cores on an integrated circuit chip that includes multiple cores.

[0080] In some implementations, the channel controller of system 100 includes at least two (optionally, three) processing stages: fetch, compute, and (optionally) flush. This may be a sequential processing pipeline that processes channel IDs in FIFO order.

[0081] The fetch process is performed using a fetch ID unit, such as fetch ID unit 702 (or 704) described below with reference to FIG. 7. In some implementations, the fetch ID unit is the first processing stage of the channel controller pipeline. The fetch ID unit receives a channel ID data structure from the processor core ID unit 114. The channel ID data encodes the length of the data to be retrieved (32 bytes), a token ID (e.g., a source address in memory 105), and a BMEM destination address. The fetch ID unit is used to load data from memory 105 and store the retrieved data at the BMEM address specified in the channel ID data. The channel ID is forwarded to the address handler unit of the channel controller, and the corresponding data is stored in the BMEM.

[0082] In some implementations, the data may be a vector of parameters stored in a buffer (e.g., a circular buffer) in the BMEM. Each time the Fetch ID unit issues a load request to the memory, the buffer write pointer is moved. In subsequent pipeline stages, as the data in the buffer is consumed, the read pointer is incremented. If the buffer does not have enough space to store the memory 105 load data in the BMEM, the Fetch ID unit may stop issuing memory 105 loads. Also, the memory 105 may generate read responses out of order. The Fetch ID unit includes response reordering logic to ensure that the channel IDs are sent to the address handler in order.

[0083] The flash process is performed using a flash ID unit, such as flash ID unit 602 (or 604) described below with reference to FIG. 6. In some implementations, the flash ID unit is an optional third processing stage in the channel controller pipeline. In general, the flash ID unit is used to move data from a source address in, for example, BMEM to a destination address in memory 105. The flash ID unit places the channel ID data structure received from the address handler into a FIFO. The channel ID data encodes the length of the data to be retrieved (32 bytes), the token ID (e.g., destination address in memory 105), and the source address in BMEM. Similar to the fetch ID unit, the flash ID unit has the reasonable responsibility of loading parameters from a source address (i.e., memory location) in BMEM and storing it to a destination address (i.e., memory location) in memory 105. Thus, the flash ID unit may be used to write the parameters of a neural network layer to any memory location in a high-bandwidth system memory via the OCI interface 500.

[0084] The OCI interface 500 must split all requests into 128-byte (128B) transfers. In some cases, requests must be split into 64B or 128B transfers depending on the address mode of the processor core implementing the request. A processor core may be configured to support multiple addressing modes. Address translation and splitting of transfers into multiple 128B (or 64B) requests is performed before sending the request to the OCI interface / crossbar 500.

[0085] The Flash ID unit updates the synchronization flag after receiving a store response from memory 105. This ensures that the load from subsequent batch processes always has the most up-to-date data. Because responses may arrive out of order, the Flash ID unit uses a simple mechanism to compare the total number of store commands sent with the total number of responses received from memory 105. If the counts match and no additional channel ID data is outstanding, the synchronization flag is updated.

[0086] FIG. 6 is an exemplary flow diagram illustrating a write operation 600 involving an OCI processor node. The write operation 600 can include one or more transactions. Each channel controller is connected to the memory 105 via an OCI interface and is used to implement a write pipeline for performing the operation 600. Each channel controller includes a flash identification (ID) unit for respectively unpacking transactions associated with the write operation 600. In the example of FIG. 6, a first channel controller 0 includes a flash ID unit 602 ("flash unit 602") and performs one or more unpacking and converting operations 606, and another second channel controller 1 includes a flash ID unit 604 ("flash unit 604") and performs one or more unpacking and converting operations 608. In some implementations, channel controller 0 and channel controller 1 are the same channel controller.

[0087] Each of the flash units 602, 604 can expand each transaction to a multiple of 128B or 64B depending on the channel operating mode of a given processor core that includes channel controllers 0 and 1, respectively. For example, as described above, an OCI transaction request encodes addressing information for retrieving data from any memory channel of the system memory using the OCI interface, or for writing data retrieved from any memory channel to a vector memory of the processor core. Thus, in some implementations, the addressing information is derived from one or more incoming OCI transaction requests received by the OCI controller. Each incoming OCI transaction request can include multiple identifiers (IDs) corresponding to an arbitrary set of memory locations of any memory channel of the system memory.

[0088] OCI transaction requests may be expanded and / or converted (e.g., split) into 64B or 128B transfers depending on the address mode of the processor core implementing the request. Also, expanding each transaction into multiples of 128B or 64B can be aligned to 32B and sent to OCI controller 502. As described above, each processor core of system 100 can operate in a channel interleaved mode or a stack interleaved mode, each of which represents an exemplary addressing (or operating) mode of the processor core.

[0089] Each OCI controller 502 can communicate with two different flash units, e.g., flash units 602, 604, or flash units corresponding to one or more other memory channels of memory 105. The controller 502 assigns an OCI ID to each write request based on the requester and transmits the write request through the corresponding write interface. For example, the controller 502 can transmit or transmit a request through a write interface connected to an exemplary channel 0 or channel 1 by selecting one of the two flash units 602, 604 in a round-robin manner using an exemplary selection logic of a channel arbiter 610. The selection logic can be based on the modified round-robin distribution algorithm 300 described above. In some implementations, each of the flash ID units 602, 604 is operable to generate a “deallocate” signal indicating completion of data processing for the channel ID. As described above, a backward pass may be performed to update the embedding table during the training phase of the neural network machine learning / data model. The deallocate signal is sent to the corresponding fetch ID unit during a parameter backward pass operation.

[0090] In some implementations, the operations performed by each of the flash units 602, 604 are not affected by the completion order of the write operations. A given flash unit 602, 604 does not need to use more than two OCI IDs for writing because it is not affected by the completion order of the write operations. Thus, the controller 502 can streamline operations and achieve computational savings or efficiency by minimizing the operation of assigning additional OCI IDs to write requests. In some implementations, the OCI interface 500 can track outstanding IDs using a bit vector. Depending on the OCI ID of the write completion, the ID is forwarded to the flash unit that generated the transaction whose outstanding number is counted.

[0091] FIG. 7 is an example flow diagram illustrating a read operation 700 involving an OCI processor node.

[0092] Each channel controller (e.g., controller 502) includes a fetch identification (ID) unit for respectively unfolding a transaction associated with a read operation 700. In the example of FIG. 7, a first channel controller 0 includes a fetch ID unit 702 ("fetch unit 702") and performs unfolding and conversion operation 706, and another second channel controller 1 includes a fetch ID unit 704 ("fetch unit 704"). For an exemplary read operation, transaction unfolding and address conversion are performed within the corresponding OCI controller 502, which then sends a request to the channel arbiter 610 to send the read request to the OCI. In this example, the OCI controller 502 corresponds to channel controller 0 or channel controller 1.

[0093] In some implementations, each fetch unit 702, 704 tracks the completion of read operations using a count tag memory and updates a sync flag memory to account for the completion. In some other implementations, the control module that spreads out the transactions is configured to generate at least one 128B transaction every four cycles to match the rate at which it can accept read data. The channel arbiter 610 services the two channel controllers in a round robin fashion and looks up available IDs in the tag tracking table 710. For example, the tracking table can hold one entry per ID and can contain 256 entries. In some implementations, the tracking table may be configured to hold more or fewer entries. In the example of 256 entries, rows 0-127 of the table may be reserved for the fetch unit 702 and rows 128-255 of the table may be reserved for the fetch unit 704.

[0094] In response to the requesting fetch unit, the channel arbiter 610 can perform a lookup to determine an area of ​​the table that contains one or more available IDs. IDs may be assigned in an incremental manner. For example, once ID0 is assigned, it is not assigned again until ID127 is assigned, even if read data for ID0 is returned. This configuration provides 128 available IDs for each channel controller in a manner that streamlines the implementation of the table and reduces or eliminates the possibility that the OCI will experience a shortage of IDs. In some implementations, each entry in the tag tracking table reserves 32B for the memory address of the lowest 32B of the transaction, reserves 13B for the BMEM address to which the data is written, reserves 2B to indicate the transaction length (a multiple of 32B), reserves 5B for count tags received from the fetch unit to track completed (possibly larger) transactions, and reserves 1B for caching to mark whether this transaction is cacheable in the read bypass buffer. In some cases, the lowest 32B of the transaction is required to be written to the read bypass buffer.

[0095] Read data from the OCI interface 500 may be sent in 1-4 cycles depending on the size of the transaction to the FIFO. Writing the first 32B line of a transaction to this FIFO triggers a lookup into the tag table to get the BMEM address, HBM address, and count tags attached to this data. The transaction may include a cache enable bit. If the cache enable bit for this transaction is set to 0 in table 710, the read bypass buffer is not updated and the ID is deallocated. If the cache bit is set to 1, the data is written to the read bypass buffer.

[0096] The read bypass buffer may be evacuated in FIFO order with the lowest 32B of a transaction written first. This, combined with the fact that the HBM address matching list is looked up in the lowest 32B of a transaction, ensures that if the lowest 32B of a transaction causes a "hit", the remaining bytes of the transaction can also be present in the buffer. Data is looked up from the fetch units 702, 704 such that accesses to a particular HBM address are the same length, so that each transaction to that particular HBM address unfolds in the same way. This implementation detail avoids a scenario where only a portion of a transaction in the read bypass buffer causes a "hit".

[0097] The memory 105 addresses from the tag table 710 may be added to the matching list and the corresponding data updated to the buffer by arbitrating the write ports of all writes from each channel OCI controller of the four channel OCI controller. To avoid the risk of evicting an entry that "hit" in the memory 105 address list, a memory address located near the next eviction candidate may be marked as "miss". In some cases, this is because there is an arbitration delay between the actual hit and the time it takes to read the data from the bypass buffer.

[0098] Depending on the architectural layout of the integrated circuit, there may be some difference in the delay between overwriting the various copies of the HBM address matching list and the bypass buffer. In some implementations, the system 100 includes a control parameter that defines a threshold that indicates how close a hit is to an evacuation candidate to be marked as a miss. To avoid the need to perform arbitration to determine whether an entry is present in the bypass buffer, the memory 105 address list may be replicated between the channel OCI controllers. The system 100 may be configured such that some (or all) entries in the read bypass buffer are invalidated between batches to ensure that the system does not use stale data. This bypass buffer invalidation request may be sent with the last ID in the batch from the fetch unit.

[0099] Read data from either the OCI or bypass buffer is sent to the BMEM on 32B lines along with the BMEM address where the data needs to be written. As described above, the corresponding fetch unit (e.g., fetch unit 702 or 704) tracks read completions using a count tag memory and updates a sync flag memory.

[0100] One or more transactions may be marked as non-cacheable and sent to another round-robin channel arbiter 610 that selects between these transactions and any cacheable transactions that had a lookup miss. The grant provided by this channel arbiter depends on the available credits indicated in the OCI interface.

[0101] Transactions marked as cacheable may be sent to a separate FIFO queue or table to look up memory 105 addresses present in the read bypass buffer. The system 100 performs address matching by searching 512 entries of the lowest 32B memory 105 addresses of the transaction, for example in parallel. In some implementations, the system 100 may include four channel OCI controllers, each controller having a copy of the memory 105 address list present in the bypass buffer shared among the four controllers. Updating the bypass buffer updates the copies of all addresses. If there is a hit in the address list, the transaction must arbitrate with transactions from the other three controllers to access the read data.

[0102] In some implementations, the OCI ID assigned to this transaction is released and the transaction is queued for read data lookup. The read data from the bypass buffer is queued for transmission back to the fetch unit that originated the request and may be taken from the most significant bits of the OCI ID originally assigned to this transaction. The transmission of this read data is done via a round robin arbiter to select between data from the OCI interface 500 and data from the bypass buffer.

[0103] Embodiments of the subject matter and functional operations described herein can be realized in digital electronic circuitry, tangible computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed in this disclosure and their structural equivalents.Embodiments of the subject matter described in this disclosure can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier and executed by or for controlling the operation of a data processing apparatus.

[0104] Alternatively or additionally, the program instructions may be encoded onto an artificially generated propagated signal, e.g., a machine-generated electrical signal generated by encoding information for transmission to an appropriate receiver for execution by a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

[0105] The term "computing system" includes all kinds of equipment, devices, and machines for processing data, such as a programmable processor, computer, or multiple processors or computers. The device may also be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the device may include code that creates an execution environment for computer programs, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0106] Computer programs (which may be referred to or described as programs, software, software applications, modules, software modules, scripts, or code) may be written in any programming language, including compiled or interpreted, declarative or procedural languages, and may be used as stand-alone programs or in any form as modules, components, subroutines, objects, or other units suitable for use within a computing environment.

[0107] A computer program can, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program involved, or in multiple co-located files (e.g., a file that stores one or more modules, subprograms or portions of code). A computer program can be implemented running on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0108] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by processing input data and generating output. The processes and logic flows may also be performed by, and the apparatus may be implemented by, special purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0109] A computer suitable for executing a computer program may be based on, by way of example, a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. Some elements of a computer are a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, and / or is operatively coupled to transmit data to and receive data from these mass storage devices. However, a computer need not have these devices. In addition, a computer may include other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0110] Computer readable media suitable for the storage of computer program instructions and data include all kinds of non-volatile memories, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0111] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer that includes a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user, a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback provided to the user may be any type of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. The computer may also interact with the user by sending and receiving documents to and from devices used by the user, such as by sending web pages to a web browser on a user client device in response to a request received from the web browser.

[0112] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, or a web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0113] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically exchange information through a communication network. The relationship of clients and servers depends upon computer programs running on the respective computers and having a client-server relationship to each other.

[0114] Although the present specification contains many specific implementation details, these details should not be considered as limiting the scope of the invention or the scope of what can be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in separate embodiments of the present disclosure may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features are described above as acting in a particular combination, notwithstanding such description, one or more features may be deleted from the described combination, or the described combination may be modified into a subcombination. Similarly, although operations are shown in the figures in a particular order, it should not be understood that these operations need to be performed in the order or sequence shown, or that all of the operations shown need to be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. For example, parallel processing may be used to execute multiple language detection methods simultaneously. Furthermore, the separation of various system elements in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program elements and systems described may generally be integrated into a single software product or packaged into multiple software products. Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Moreover, the processes depicted in the accompanying figures do not necessarily have to follow the particular order or sequence depicted to achieve desirable results. In certain implementations, multi-tasking and parallel processing may be advantageous.

Claims

1. 1. An integrated circuit configured to accelerate machine learning computations, comprising: the integrated circuit comprises a plurality of processor cores; Each processor core is A plurality of channel controllers; an interface controller configured to connect each channel controller of the plurality of channel controllers to a respective memory channel of a system memory; a respective fetch unit in each channel controller of the plurality of channel controllers; an integrated circuit configured to: i) receive a request encoding addressing information; ii) retrieve data from any memory channel of the system memory based on the addressing information using the interface controller; and iii) write the data retrieved from any memory channel to a vector memory of the processor core via the channel controller that includes the fetch unit.

2. the interface controller is operable to control an in-memory OCI node; 2. The integrated circuit of claim 1, wherein the in-memory OCI node includes: i) a plurality of read interfaces used to retrieve data from any memory location along any memory channel of the system memory; and ii) a plurality of write interfaces used to write data to any memory location along any memory channel of the system memory.

3. 3. The integrated circuit of claim 2, wherein the in-memory OCI node includes a first-in-first-out (FIFO) memory queue configured to store incoming OCI transaction requests during arbitration of an existing OCI transaction that: i) retrieves data from any memory location in the system memory; or ii) writes data to any memory location in the system memory.

4. 4. The integrated circuit of claim 3, wherein the interface controller is configured to receive a request to encode the addressing information, generate a plurality of control signals based on a plurality of IDs in the addressing information, and provide the plurality of control signals to any memory channel of the system memory to fetch data stored in a memory location of the system memory.

5. the addressing information is derived from one or more incoming OCI transaction requests received by the interface controller; 5. The integrated circuit of claim 4, wherein at least one existing incoming OCI transaction request to access any memory location of the system memory is pre-stored in the first-in-first-out (FIFO) memory queue.

6. 6. The integrated circuit of claim 5, wherein each incoming OCI transaction request includes a number of identifiers (IDs) corresponding to any set of memory locations in any memory channel of the system memory.

7. the interface controller includes two or more addressing modes; The integrated circuit of claim 6 , wherein the interface controller is operable to generate control signals to enable selection between the two or more addressing modes of the interface controller.

8. 8. The integrated circuit of claim 7, wherein a first addressing mode of the two or more addressing modes is a channel interleave mode that constrains mapping of embedded tables to memory channels of the system memory.

9. 9. The integrated circuit of claim 8, wherein a second addressing mode of the two or more addressing modes is a stack interleave mode that extends the mapping of the embedded table to memory channels of the system memory.

10. 8. The integrated circuit of claim 7, wherein each incoming OCI transaction request is received by the fetch unit and processed by the interface controller to initiate access to any memory location indicated in the incoming OCI transaction request.

11. 11. The integrated circuit of claim 10, wherein each incoming OCI transaction request to access data stored in the system memory encodes i) a 32-byte length corresponding to the data being accessed, ii) a token ID representing a unique source address of any memory location within the system memory, and iii) a destination address of a memory location in the vector memory.

12. the embedding table is partitioned into a number of data shards that are allocated across memory channels across the system memory; 10. The integrated circuit of claim 9, wherein the interface controller selects the stacked interleaved mode to access any portion of the embedded table with any of the plurality of channel controllers.

13. a respective flash unit in each channel controller of the plurality of channel controllers; 2. The integrated circuit of claim 1, wherein each said flash unit is configured to: i) receive a request encoding addressing information; and ii) use said interface controller to write data from a source address in said vector memory to any memory location in said system memory based on said addressing information.

14. The machine learning computations are performed to process neural network inputs through neural network layers; 14. The integrated circuit of claim 13, wherein the flash unit is used to write the parameters of the neural network layers to any memory location in the system memory with the interface controller.

15. 1. A computer-implemented method performed with an integrated circuit configured to accelerate machine learning computations, the integrated circuit comprising a system memory and a processor core including a plurality of channel controllers, the method comprising: receiving a plurality of requests, each request of the plurality of requests encoding addressing information; For a first request of the plurality of requests, identifying any memory location in any memory channel of the system memory based on addressing information of the first request; obtaining data from any memory location in the system memory using an on-chip interconnect (OCI) interface for connecting each channel controller of the plurality of channel controllers to a respective memory channel of the system memory; and writing the data obtained from the memory location to a vector memory of the processor core based on the addressing information of the first request.

16. the on-chip interconnect interface comprises an in-memory OCI node including a plurality of read interfaces and a plurality of write interfaces; The method comprises: retrieving data from any memory location along any memory channel of the system memory using the plurality of read interfaces based on control signals generated by the on-chip interconnect interface; 16. The method of claim 15, further comprising: writing data with the multiple write interfaces to any memory location along any memory channel of the system memory based on control signals generated by the on-chip interconnect interface.

17. the in-memory OCI node comprises a first-in, first-out (FIFO) memory queue; 17. The method of claim 16, further comprising storing an incoming OCI transaction request in the first-in-first-out (FIFO) memory queue during arbitration of an existing OCI transaction that: i) retrieves data from any memory location in the system memory; or ii) writes data to any memory location in the system memory.

18. The on-chip interconnect interface receiving the request encoding the addressing information; and the on-chip interconnect interface generating a plurality of control signals based on a plurality of IDs in the addressing information; 20. The method of claim 17, further comprising: the on-chip interconnect interface providing the plurality of control signals to any memory channel of the system memory to fetch data stored in a memory location of the system memory.

19. the addressing information is derived from one or more incoming OCI transaction requests received by the on-chip interconnect interface; 20. The method of claim 18, wherein at least one existing incoming OCI transaction request to access any memory location of the system memory is pre-stored in the first-in-first-out (FIFO) memory queue.

20. 20. The method of claim 19, wherein each incoming OCI transaction request includes a number of identifiers (IDs) corresponding to a given set of memory locations in a given memory channel of the system memory.

Images