Integrated circuit device with deep learning accelerator and random access memory

Abstract

Systems, apparatuses, and methods related to deep learning accelerators and memories are described. An integrated circuit is configured to perform computations on at least matrix operands and is configured to include: a random access memory configured to store instructions executable by the deep learning accelerator and to store matrices of an artificial neural network; a connection between the random access memory and the deep learning accelerator; and an interface to a memory controller. The interface is configured to facilitate access to the random access memory by the memory controller. In response to an instruction provided in the random access memory, the deep learning accelerator can execute the instruction to apply input stored in the random access memory to the artificial neural network, generate an output from the artificial neural network, and store the output in the random access memory.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 844,988, filed April 9, 2020, entitled “Integrated Circuit Device with Deep Learning Accelerator and Random Access Memory,” the entire disclosure of which is hereby incorporated herein by reference. Technical Field

[0003] At least some of the embodiments disclosed herein generally relate to integrated circuit devices, and more specifically, but not limited to, integrated circuit devices having accelerators for artificial neural networks (ANNs), such as ANNs configured through machine learning and / or deep learning. Background Technology

[0004] Artificial neural networks (ANNs) use a network of neurons to process inputs to the network and produce outputs from the network.

[0005] For example, each neuron in a network receives a set of inputs. Some of the inputs to a neuron may be the outputs of other neurons in the network; and some of the inputs to a neuron may be inputs provided to the neural network. The input / output relationships among neurons in a network represent the neuronal connectivity within the network.

[0006] For example, each neuron can have a bias, an activation function, and a set of synaptic weights for its input. The activation function can be a step function, a linear function, a log-sigmoid function, etc. Different neurons in the network can have different activation functions.

[0007] For example, each neuron can produce a weighted sum of its input and its bias, and then produce an output as a function of the weighted sum, which is calculated using the neuron's activation function.

[0008] The relationship between the inputs and outputs of an ANN is generally defined by the ANN model, which contains data representing the connectivity of neurons in the network, as well as the biases, activation functions, and synaptic weights of each neuron. Based on a given ANN model, a computing device can be configured to compute the network's output from a given set of inputs to the network.

[0009] For example, the input to the ANN network can be generated based on camera input; and the output from the ANN network can be the recognition of items such as events or objects.

[0010] Generally, supervised methods can be used to train ANNs, where the parameters in the ANN are tuned to minimize or reduce the error between a known output associated with or produced by a given input and a computed output produced by applying the input to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.

[0011] Alternatively or in combination, unsupervised methods can be used to train ANNs, where the exact output produced by a given set of inputs is unknown until training is complete. Trainable ANNs can classify items into multiple categories or data points into clusters.

[0012] A variety of training algorithms can be used for complex machine learning / training paradigms.

[0013] Deep learning uses multiple layers of machine learning to progressively extract features from input data. For example, lower layers can be configured to identify edges in an image; and higher layers can be configured to identify captured items in the image, such as faces, objects, events, etc., based on the edges detected by the lower layers. Deep learning can be implemented via artificial neural networks (ANNs), such as deep neural networks, deep belief networks, recurrent neural networks, and / or convolutional neural networks.

[0014] Deep learning has been applied in many fields, such as computer vision, speech / audio recognition, natural language processing, machine translation, bioinformatics, drug design, medical image processing, and games. Attached Figure Description

[0015] The embodiments are illustrated by way of example and not by way of limitation in the figures, in which similar references indicate similar elements.

[0016] Figure 1 An integrated circuit device having a configured deep learning accelerator and random access memory is shown according to one embodiment.

[0017] Figure 2 A processing unit configured to perform matrix-matrix operations according to one embodiment is shown.

[0018] Figure 3 A processing unit configured to perform matrix-vector operations according to one embodiment is shown.

[0019] Figure 4 A processing unit configured to perform vector-vector operations according to one embodiment is shown.

[0020] Figure 5 A deep learning accelerator and random access memory configured to autonomously apply inputs to a trained artificial neural network, according to one embodiment, are shown.

[0021] Figure 6 A method implemented in an integrated circuit device according to one embodiment is shown. Detailed Implementation

[0022] At least some embodiments disclosed herein provide a general-purpose integrated circuit device configured to perform artificial neural network (ANN) computations with reduced power consumption and computation time. The integrated circuit device includes a deep learning accelerator (DLA) and random access memory.

[0023] A deep learning accelerator (DLA) contains a set of general-purpose programmable hardware computation logic that is specialized and / or optimized to perform parallel vector and / or matrix computations, including but not limited to vector and / or matrix multiplication and accumulation.

[0024] In addition, a deep learning accelerator (DLA) may contain one or more arithmetic logic units (ALUs) to perform arithmetic and bitwise operations on integer binary numbers.

[0025] Deep learning accelerators (DLAs) can be programmed via a set of instructions to perform computations on artificial neural networks (ANNs).

[0026] The granularity of a Deep Learning Accelerator (DLA) for vector and matrix operations corresponds to the largest unit of vector / matrix that can be operated on during the execution of an instruction by the DLA. During instructions performing predefined operations on vector / matrix operands, the DLA can operate on the elements of the vector / matrix operands in parallel to reduce execution time and / or energy consumption associated with memory / data access. Operations on vector / matrix operands at the granularity of the DLA can be used as building blocks to perform computations on larger vectors / matrixes.

[0027] The implementation of a typical / practical artificial neural network (ANN) involves vector / matrix operands with a size larger than the operational granularity of a deep learning accelerator (DLA). To implement such an ANN using a DLA, computations involving large-sized vector / matrix operands can be decomposed into computations of vector / matrix operands at the granularity of the DLA. A DLA can be programmed with instructions to perform computations involving large vector / matrix operands. For example, the atomic computational capabilities of a DLA in manipulating vectors and matrices at the granularity of the DLA in response to instructions can be programmed to perform computations within an artificial neural network (ANN).

[0028] In some implementations, a deep learning accelerator (DLA) does not possess some of the logical computational capabilities of a typical central processing unit (CPU). However, a deep learning accelerator (DLA) can be configured with sufficient logic units to process the input data provided to an artificial neural network (ANN) and produce the output of the ANN according to a set of instructions generated for the deep learning accelerator (DLA). Therefore, a deep learning accelerator (DLA) can perform ANN computations with minimal or no assistance from a central processing unit (CPU) or another processor. Optionally, a conventional general-purpose processor can also be configured as a portion of the deep learning accelerator (DLA) to perform operations that cannot be efficiently implemented using the vector / matrix processing unit of the deep learning accelerator (DLA), and / or operations that cannot be performed by the vector / matrix processing unit of the deep learning accelerator (DLA).

[0029] A typical artificial neural network (ANN) can be described / specified in a standard format (e.g., Open Neural Network Exchange (ONNX)). A compiler can be used to translate the description of an ANN into a set of instructions for a deep learning accelerator (DLA) to perform computations on the ANN. The compiler can optimize this set of instructions to improve the performance of the DLA when implementing the ANN.

[0030] Deep learning accelerators (DLAs) may have local memory, such as registers, buffers, and / or caches, configured to store vector / matrix operands and the results of vector / matrix operations. Intermediate results in registers can be pipelined / shifted within the DLA as operands for subsequent vector / matrix operations, reducing time and energy consumption in accessing memory / data and thus accelerating the typical pattern of vector / matrix operations when implementing typical artificial neural networks (ANNs). The capacity of registers, buffers, and / or caches in a DLA is typically insufficient to hold the entire dataset used for computations implementing typical ANNs. Therefore, random access memory coupled to the DLA is configured to provide improved data storage capabilities for implementing typical ANNs. For example, the DLA loads data and instructions from random access memory and stores the results back into random access memory.

[0031] The communication bandwidth between the Deep Learning Accelerator (DLA) and random access memory (RAM) is configured to optimize or maximize the utilization of the DLA's computational power. For example, high communication bandwidth between the DLA and RAM can be provided to allow vector / matrix operands to be loaded from RAM into the DLA and stored back into RAM within a timeframe approximately equal to the DLA's processing time, enabling computation on the vector / matrix operands to be performed. The granularity of the DLA can be configured to increase the ratio between the amount of computation performed by the DLA and the size of the vector / matrix operands, thereby reducing data access traffic between the DLA and RAM. This reduces the bandwidth requirements for communication between the DLA and RAM. Therefore, bottlenecks in data / memory access can be reduced or eliminated.

[0032] Figure 1 An integrated circuit device (101) with a configured deep learning accelerator (103) and random access memory (105) is shown according to one embodiment.

[0033] exist Figure 1The deep learning accelerator (103) includes a processing unit (111), a control unit (113), and local memory (115). When vector and matrix operands are in local memory (115), the control unit (113) can use the processing unit (111) to perform vector and matrix operations according to instructions. In addition, the control unit (113) can load instructions and operands from random access memory (105) via a memory interface (117) and a high-speed / bandwidth connection (119).

[0034] The integrated circuit device (101) is configured to be enclosed within an integrated circuit package using the pins or contacts of the memory controller interface (107).

[0035] The memory controller interface (107) is configured to support standard memory access protocols, such that the integrated circuit device (101) behaves as a typical memory controller (e.g., 108 of connector 106 connected to the integrated circuit device (101)) in the same way as a conventional random access memory device without a deep learning accelerator (DLA) (103). For example, a memory controller external to the integrated circuit device (101) can access the random access memory (105) in the integrated circuit device (101) via the memory controller interface (107) using standard memory access protocols.

[0036] The integrated circuit device (101) is configured with a high-bandwidth connection (119) between a random access memory (105) enclosed within the integrated circuit device (101) and a deep learning accelerator (DLA) (103). The bandwidth of the connection (119) is higher than the bandwidth of the connection (109) between the random access memory (105) and the memory controller interface (107).

[0037] In one embodiment, both the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via the same set of buses or wires. Therefore, the bandwidth used to access the random access memory (105) is shared between the memory interface (117) and the memory controller interface (107). Alternatively, the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via a separate set of buses or wires. Optionally, the random access memory (105) may comprise multiple segments that can be accessed simultaneously via a connection (119). For example, while the memory interface (117) is accessing a segment of the random access memory (105), the memory controller interface (107) may simultaneously access another segment of the random access memory (105). For example, different segments can be configured on different planes / rows of different integrated circuit dies and / or memory cells; and different segments can be accessed in parallel to increase the throughput when accessing random access memory (105). For example, the memory controller interface (107) is configured to access a data cell of a predetermined size at a time; and the memory interface (117) is configured to access multiple data cells at a time, each data cell being the same predetermined size.

[0038] In one embodiment, the random access memory (105) and the integrated circuit device (101) are disposed on different integrated circuit dies disposed within the same integrated circuit package. Furthermore, the random access memory (105) may be disposed on one or more integrated circuit dies that allow simultaneous parallel access to multiple data elements.

[0039] In some implementations, the number of data elements of a vector or matrix that can be accessed in parallel via connection (119) corresponds to the granularity of the deep learning accelerator (DLA) operating on the vector or matrix. For example, when the processing unit (111) can operate on several vector / matrix elements in parallel, connection (119) is configured to load or store the same number or a multiple thereof of elements in parallel via connection (119).

[0040] Optionally, the data access speed of the connection (119) can be configured based on the processing speed of the deep learning accelerator (DLA) (103). For example, after a certain amount of data and instructions have been loaded into the local memory (115), the control unit (113) can use the processing unit (111) to execute instructions to manipulate the data to produce output. During the processing time used to produce the output, the access bandwidth of the connection (119) allows the same amount of data and instructions to be loaded into the local memory (115) for the next operation, and the same amount of output to be stored back into the random access memory (105). For example, while the control unit (113) uses a portion of the local memory (115) to process data and produce output, the memory interface (117) can unload the output of the previous operation from another portion of the local memory (115) into the random access memory (105), and load operand data and instructions into the other portion of the local memory. Therefore, the utilization and performance of the deep learning accelerator (DLA) are not limited or reduced by the bandwidth of the connection (119).

[0041] Random access memory (105) can be used to store model data of the artificial neural network (ANN) and to buffer input data of the artificial neural network (ANN). The model data is not changed frequently. The model data may contain output generated by a compiler for a deep learning accelerator (DLA) to implement the artificial neural network (ANN). The model data typically contains matrices used in the description of the artificial neural network (ANN) and instructions generated for the deep learning accelerator (DLA) (103) to perform vector / matrix operations of the artificial neural network (ANN) at a granularity based on the deep learning accelerator (DLA) (103). The instructions operate not only on the vector / matrix operations of the artificial neural network (ANN) but also on the input data of the artificial neural network (ANN).

[0042] In one embodiment, when input data is loaded or updated in random access memory (105), the control unit (113) of the deep learning accelerator (DLA) (103) can automatically execute instructions of the artificial neural network (ANN) to produce the output of the ANN. The output is stored in a predefined area in random access memory (105). The deep learning accelerator (DLA) (103) can execute instructions without assistance from the central processing unit (CPU). Therefore, communication for coordination between the deep learning accelerator (DLA) (103) and a processor (e.g., a central processing unit (CPU)) outside the integrated circuit device (101) can be reduced or eliminated.

[0043] Optionally, the logic circuitry of the deep learning accelerator (DLA) (103) can be implemented via complementary metal-oxide-semiconductor (CMOS). For example, CMOS technology under the array of memory cells (CUA) of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103), including a processing unit (111) and a control unit (113). Alternatively, CMOS technology in the memory cell array of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103).

[0044] In some implementations, the deep learning accelerator (DLA) (103) and random access memory (105) may be implemented on a separate integrated circuit die and connected using through-silicon vias (TSVs) to increase the data bandwidth between the deep learning accelerator (DLA) (103) and the random access memory (105). For example, the deep learning accelerator (DLA) (103) may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0045] Alternatively, the deep learning accelerator (DLA) (103) and random access memory (105) can be configured in a separate integrated circuit package and connected via multiple point-to-point connections on a printed circuit board (PCB) for parallel communication, thereby increasing data transmission bandwidth.

[0046] Random access memory (105) can be volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase-change memory (PCM), magnetic RAM (MRAM), resistive random access memory, cross-point memory devices, and memory devices. Cross-point memory devices can use transistorless memory elements, each of which has memory cells and selectors stacked together in a column. The column of memory elements is connected via two layers of wiring laid in a vertical direction, wherein the wiring of one layer is laid in one direction above the column of memory elements, and the wiring of the other layer is laid in the other direction and below the column of memory elements. Each memory element can be selected individually at the intersection of a line on each of the two layers. Cross-point memory devices are fast and non-volatile and can be used as a general-purpose memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM). Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0047] For example, non-volatile memory can be configured to implement at least a portion of random access memory (105). The non-volatile memory within random access memory (105) can be used to store model data of an artificial neural network (ANN). Therefore, after the integrated circuit device (101) is powered off and restarted, it is not necessary to reload the model data of the artificial neural network (ANN) into the integrated circuit device (101). Furthermore, the non-volatile memory can be programmable / rewritable. Therefore, the model data of the artificial neural network (ANN) in the integrated circuit device (101) can be updated or replaced to implement an updated artificial neural network (ANN) or another artificial neural network (ANN).

[0048] The processing unit (111) of the deep learning accelerator (DLA) (103) may include vector-vector units, matrix-vector units, and / or matrix-matrix units. The following is in conjunction with... Figures 2 to 4 Discuss instances of units configured to perform vector-vector operations, matrix-vector operations, and matrix-matrix operations.

[0049] Figure 2 A processing unit (121) configured to perform matrix-matrix operations according to one embodiment is shown. For example, Figure 2 The matrix-matrix unit (121) can be used as Figure 1One of the processing units (111) of the Deep Learning Accelerator (DLA) (103).

[0050] exist Figure 2 In this configuration, the matrix-matrix unit (121) comprises multiple kernel buffers (131 to 133) and multiple map groups (151 to 153). Each of the map groups (151 to 153) stores a vector of a matrix operand, which has multiple vectors stored in the map groups (151 to 153); and each of the kernel buffers (131 to 133) stores a vector of another matrix operand, which has multiple vectors stored in the kernel buffers (131 to 133). The matrix-matrix unit (121) is configured to perform multiplication and accumulation operations on the elements of the two matrix operands using multiple matrix-vector units (141 to 143) operating in parallel.

[0051] A crossbar switch (123) connects the mapping groups (151 to 153) to the matrix-vector units (141 to 143). Identical matrix operands stored in the mapping groups (151 to 153) are provided to each of the matrix-vector units (141 to 143) via the crossbar switch (123); and the matrix-vector units (141 to 143) receive data elements from the mapping groups (151 to 153) in parallel. Each of the kernel buffers (131 to 133) is connected to the corresponding matrix-vector unit (141 to 143) and provides a vector operand to the corresponding matrix-vector unit. The matrix-vector units (141 to 143) simultaneously operate to compute the operation of multiplying the identical matrix operands stored in the mapping groups (151 to 153) by the corresponding vector stored in the kernel buffers (131 to 133). For example, the matrix-vector unit (141) performs multiplication on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (131), while the matrix-vector unit (143) simultaneously performs multiplication on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (133).

[0052] exist Figure 2 Each of the matrix-vector units (141 to 143) in the matrix can be like... Figure 3 Implemented as shown.

[0053] Figure 3 A processing unit (141) configured to perform matrix-vector operations according to one embodiment is shown. For example, Figure 3 The matrix-vector unit (141) can be used as Figure 2Any one of the matrix-vector units in the matrix-matrix unit (121).

[0054] exist Figure 3 In the diagram, each of the mapping groups (151 to 153) is arranged in a manner similar to... Figure 2 A vector of matrix operands is stored in a mapping group (151 to 153), wherein the matrix operands have multiple vectors stored in the mapping group (151 to 153). Figure 3 The crossbar switch (123) in the middle provides vectors from the mapping group (151) to the vector-vector units (161 to 163). The same vectors stored in the kernel buffer (131) are provided to the vector-vector units (161 to 163).

[0055] Vector-vector units (161 to 163) operate simultaneously to compute the operation of multiplying the corresponding vector operands stored in the map arrays (151 to 153) by the same vector operands stored in the kernel buffer (131). For example, vector-vector unit (161) performs a multiplication operation on the vector operands stored in the map array (151) and the vector operands stored in the kernel buffer (131), while vector-vector unit (163) simultaneously performs a multiplication operation on the vector operands stored in the map array (153) and the vector operands stored in the kernel buffer (131).

[0056] when Figure 3 The matrix-vector unit (141) is implemented in Figure 2 When in the matrix-matrix unit (121), the matrix-vector unit (141) can use the mapping group (151 to 153), the cross switch (123) and the kernel buffer (131) of the matrix-matrix unit (121).

[0057] Figure 3 Each of the vector-vector units (161 to 163) in the vector array can be as follows: Figure 4 Implement it in the manner described herein.

[0058] Figure 4 A processing unit (161) configured to perform vector-vector operations according to one embodiment is shown. For example, Figure 4 The vector-vector unit (161) can be used as Figure 3 Any one of the vector-vector units in the matrix-vector unit (141).

[0059] exist Figure 4In this, the vector-vector unit (161) has multiple multiply-accumulate (MAC) units (171 to 173). Each of the multiply-accumulate (MAC) units (171 to 173) can receive two numbers as operands, perform multiplication of the two numbers, and add the result of the multiplication to the sum maintained in the multiply-accumulate (MAC) unit.

[0060] Each of the vector buffers (181 and 183) stores a list of numbers. A pair of numbers, each from one of the vector buffers (181 and 183), can be provided as input to each of the multiply-accumulate (MAC) units (171 to 173). The multiply-accumulate (MAC) units (171 to 173) can receive multiple pairs of numbers from the vector buffers (181 and 183) in parallel and perform the multiply-accumulate (MAC) operation in parallel. The outputs from the multiply-accumulate (MAC) units (171 to 173) are stored in a shift register (175); and the accumulator (177) calculates the sum of the results in the shift register (175).

[0061] when Figure 4 The vector-vector unit (161) is implemented in Figure 3 When in the matrix-vector unit (141), the vector-vector unit (161) can use a mapping group (e.g., 151 or 153) as a vector buffer (181) and use the kernel buffer (131) of the matrix-vector unit (141) as another vector buffer (183).

[0062] Vector buffers (181 and 183) may have the same length to store the same number / count of data elements. The length may be equal to or a multiple of the count of the multiply-accumulate (MAC) units (171 to 173) in the vector-vector unit (161). When the length of the vector buffers (181 and 183) is a multiple of the count of the multiply-accumulate (MAC) units (171 to 173), input pairs equal to the number of counts of the multiply-accumulate (MAC) units (171 to 173) may be provided as input from the vector buffers (181 and 183) to the multiply-accumulate (MAC) units (171 to 173) in each iteration; and the vector buffers (181 and 183) feed their elements into the multiply-accumulate (MAC) units (171 to 173) over multiple iterations.

[0063] In one embodiment, the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and the random access memory (105) is sufficient for the matrix-matrix unit (121) to use a portion of the random access memory (105) as a map group (151 to 153) and a kernel buffer (131 to 133).

[0064] In another embodiment, the mapping groups (151 to 153) and kernel buffers (131 to 133) are implemented in a portion of the local memory (115) of the deep learning accelerator (DLA) (103). While the matrix-matrix unit (121) is performing computations using the mapping groups (151 to 153) and kernel buffers (131 to 133) implemented in different portions of the local memory (115) of the deep learning accelerator (DLA) (103), the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and random access memory (105) is sufficient to load the matrix operands of the next operating cycle of the matrix-matrix unit (121) into another portion of the local memory (115).

[0065] Figure 5 A deep learning accelerator and random access memory configured to autonomously apply inputs to a trained artificial neural network, according to one embodiment, are shown.

[0066] Artificial neural networks (ANNs) (201) trained by machine learning (e.g., deep learning) can be described using standard formats (e.g., Open Neural Network Exchange (ONNX)). Describing a trained ANN (201) using standard formats identifies the properties of artificial neurons and their connectivity.

[0067] exist Figure 5 In this process, the deep learning accelerator (DLA) compiler (203) transforms the trained ANN (201) by generating instructions (205) for the deep learning accelerator (DLA) (103) and a matrix (207) corresponding to the properties of the artificial neurons and their connectivity. The instructions (205) and the matrix (207) generated by the DLA compiler (203) from the trained ANN (201) can be stored in the random access memory (105) for the deep learning accelerator (DLA) (103).

[0068] For example, random access memory (105) and deep learning accelerator (DLA) (103) can be used as follows: Figure 1 The integrated circuit device (101) is connected via a high-bandwidth connection (119). Figure 5 Autonomous computation based on instructions (205) and matrices (207) can be implemented in Figure 1 In the integrated circuit device (101). Alternatively, the random access memory (105) and the deep learning accelerator (DLA) (103) may be configured on a printed circuit board having multiple point-to-point serial buses with parallel extensions for connection (119).

[0069] exist Figure 5In the process, after the results of the DLA compiler (203) are stored in random access memory (105), the application of the trained ANN (201) to process the input (211) to the trained ANN (201) to produce the corresponding output (213) of the trained ANN (201) can be triggered by the presence of the input (211) in random access memory (105) or by another indication provided in random access memory (105).

[0070] In response, the deep learning accelerator (DLA) (103) executes instructions (205) to combine the input (211) and the matrix (207). The execution of instructions (205) may involve generating a mapping matrix by a grouping (151 to 153) of one or more matrix-matrix units (e.g., 121) of the deep learning accelerator (DLA) (103) of the matrix.

[0071] In some embodiments, the input to the ANN (211) is in the form of an initial mapping matrix. A portion of the initial mapping matrix can be retrieved from random access memory (105) as a matrix operand stored in a mapping array (151 to 153) of matrix-matrix units (121). Alternatively, the DLA instructions (205) also include instructions for the deep learning accelerator (DLA) (103) to generate an initial mapping matrix from the input (211).

[0072] According to the DLA instruction (205), the deep learning accelerator (DLA) (103) loads matrix operands into the kernel buffers (131 to 133) and map groups (151 to 153) of its matrix-matrix unit (121). The matrix-matrix unit (121) performs matrix computations on the matrix operands. For example, the DLA instruction (205) decomposes the matrix computations of the trained ANN (201) according to the computational granularity of the deep learning accelerator (DLA) (103) (e.g., the size / dimension of the matrix loaded in the matrix-matrix unit (121) as a matrix operand), and applies the input feature maps to the kernel of one layer of artificial neurons to produce outputs as inputs for the next layer of artificial neurons.

[0073] After the computation of the trained ANN (201) executed according to the instruction (205) is completed, the deep learning accelerator (DLA) (103) immediately stores the output (213) of the ANN (201) in a predefined location in random access memory (105) or in a location provided in random access memory (105) as specified in the instruction to trigger the computation.

[0074] when Figure 5 The technology implemented in Figure 1When the input (211) is in the integrated circuit device (101), an external device connected to the memory controller interface (107) can write the input (211) into the random access memory (105) and trigger the deep learning accelerator (DLA) (103) to apply the input (211) to the autonomous computation of the trained ANN (201). After a certain time period, the output (213) is available in the random access memory (105); and the external device can read the output (213) via the memory controller interface (107) of the integrated circuit device (101).

[0075] For example, a predefined location in random access memory (105) can be configured to store an instruction that triggers the autonomous execution of an instruction (205) by a deep learning accelerator (DLA) (103). The instruction may optionally include the location of an input (211) within random access memory (105). Thus, during the autonomous execution of an instruction (205) that processes the input (211), an external device can retrieve outputs generated during a previous run of the instruction (205) and / or store another set of inputs for the next run of the instruction (205).

[0076] Optionally, another predefined location in the random access memory (105) may be configured to store an indication of the current progress status of the instruction (205). Furthermore, the indication may include a prediction of the completion time of the current execution of the instruction (205) (e.g., estimated based on previous executions of the instruction (205)). Thus, an external device can check the completion status within a suitable time window to retrieve the output (213).

[0077] In some embodiments, the random access memory (105) is configured with sufficient capacity to store multiple sets of inputs (e.g., 211) and outputs (e.g., 213). Each set can be configured in the random access memory (105) in a predetermined time slot / region.

[0078] The deep learning accelerator (DLA) (103) can autonomously execute instructions (205) to produce an output (213) from an input (211) based on a matrix (207) stored in random access memory (105) without the assistance of a processor or device located outside the integrated circuit device (101).

[0079] Figure 6 A method implemented in an integrated circuit device according to one embodiment is illustrated. For example, Figure 6 The method can be implemented in Figure 1 Integrated circuit device (101) or similar Figure 5 It is shown in another device, but configured on a printed circuit board.

[0080] At block 301, the random access memory (105) of the computing device (e.g., 101) can be accessed via an interface (107) to the memory controller. The computing device (e.g., 101) may have a processing unit (e.g., 111) configured to perform computations on at least matrix operands, such as matrix operands stored in a map array (151 to 153) and matrix operands stored in a kernel buffer (131 to 133).

[0081] For example, a computing device (e.g., 101) may be enclosed within an integrated circuit package; and a set of connections may connect an interface (107) to a memory controller located outside the integrated circuit package.

[0082] At block 303, instructions (205) executable by a processing unit (e.g., 111) can be written into random access memory (105) via interface (107).

[0083] At box 305, the matrix (207) of the artificial neural network (201) can be written into the random access memory (105) via the interface (107). The matrix (207) identifies the properties and / or state of the artificial neural network (201).

[0084] Optionally, at least a portion of the random access memory (105) is non-volatile and configured to store instructions (205) and matrices (07) of the artificial neural network (201).

[0085] At box 307, the first input (211) to the artificial neural network can be written to random access memory (105) via interface (107).

[0086] At block 309, an instruction is provided in random access memory (105) to cause processing unit (111) to begin execution of instruction (205). In response to the instruction, processing unit (111) executes instruction to combine first input (211) with matrix (207) of artificial neural network (201) to generate first output (213) from artificial neural network (201) and stores first output (213) in random access memory (105).

[0087] For example, the indication may be the address of a first input (211) in random access memory (105); and the indication may be stored at a predetermined location in random access memory (105) to initiate the execution of an instruction (205) for the input (211) identified by the address. Optionally, the indication may also include an address for storing an output (213).

[0088] At box 311, the first output (213) can be read from random access memory (105) via interface (107).

[0089] For example, a computing device (e.g., 101) may have a deep learning accelerator (103) formed on a first integrated circuit die and random access memory (105) formed on one or more second integrated circuit dies. The connection (119) between the first integrated circuit die and the one or more second integrated circuit dies may include through-silicon vias (TSVs) to provide high bandwidth for memory access.

[0090] For example, a compiler (203) can be used to convert the description of the artificial neural network (201) into instructions (205) and matrices (207). The combination of the instructions (205) and matrices (207) stored in random access memory (105) and the deep learning accelerator (103) provides an autonomous implementation of the artificial neural network (201) that can automatically convert the input (211) to the artificial neural network (201) into its output (213).

[0091] For example, during the time period in which the deep learning accelerator (103) executes instructions (205) to generate a first output (213) from a first input (211) according to the matrix (207) of the artificial neural network (201), a second input to the artificial neural network (201) can be written to an alternative location in the random access memory (105) via the interface (107). After the first output (213) is stored in the random access memory (105), an instruction can be provided in the random access memory to cause the deep learning accelerator (103) to resume execution of instructions and generate a second output from the second input.

[0092] During the time period in which the deep learning accelerator (103) executes instructions (205) to generate a second output from a second input according to the matrix (207) of the artificial neural network (201), the first output (213) can be read from the random access memory (105) via the interface (107); and another input can be written to the random access memory in place of the first input (211) or written to a different location. The process can be repeated for a series of inputs.

[0093] A deep learning accelerator (103) may include at least one matrix-matrix unit (121) capable of executing instructions on two matrix operands. The two matrix operands may be a first matrix and a second matrix. Each of the two matrices has multiple vectors. The matrix-matrix unit (121) may include multiple matrix-vector units (141 to 143) configured to operate in parallel. Each of the matrix-vector units (141 to 143) is configured to operate on the first matrix and a vector from the second matrix in parallel with other matrix-vector units. Furthermore, each of the matrix-vector units (141 to 143) may have multiple vector-vector units (161 to 163) configured to operate in parallel. Each of the vector-vector units (161 to 163) is configured to operate on a vector from the first matrix and a common vector operand of the corresponding matrix-vector unit in parallel with other vector-vector units. In addition, each of the vector-vector units (161 to 163) may have multiple multiply-accumulate units (171 to 173) configured to operate in parallel.

[0094] In addition to the processing unit (111), the deep learning accelerator (103) may also have local memory (115) and a control unit (113). The control unit (113) may load instructions (205) and matrix operands (e.g., 207) from random access memory (105) for execution by the processing unit (111). The local memory may cache matrix operands used by the matrix-matrix unit. The connection (119) may be configured with bandwidth sufficient to load a set of matrix operands from random access memory (105) into local memory (115) during the time period in which the matrix-matrix unit performs operations on two other matrix operands. Furthermore, during the time period, the bandwidth is sufficient to store the results produced by the matrix-matrix unit (121) in the previous instruction execution from local memory (115) into random access memory (105).

[0095] This disclosure includes methods and apparatus for performing the methods described above, a data processing system for performing these methods, and a computer-readable medium containing instructions that, when executed on the data processing system, cause the system to perform these methods.

[0096] A typical data processing system may contain interconnects (such as buses and system core logic) that interconnect microprocessors and memory. Microprocessors are typically coupled to cache memory.

[0097] Interconnectors link one or more microprocessors and memories together, and also interconnect them to one or more I / O devices via one or more input / output (I / O) controllers. I / O devices may include display devices and / or peripheral devices such as mice, keyboards, modems, network interfaces, printers, scanners, cameras, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices (e.g., printers, scanners, mice, and / or keyboards) are optional.

[0098] Interconnectors may include one or more buses that are connected to each other via various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling USB (Universal Serial Bus) peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0099] Memory can include one or more of the following: ROM (Read-Only Memory), volatile RAM (Random Access Memory), and non-volatile memory such as hard disk drives and flash memory.

[0100] Volatile RAM is typically implemented as dynamic RAM (DRAM) that requires continuous power to refresh or maintain data in memory. Non-volatile memory is typically magnetic hard disk drives, magnetic optical disk drives, optical disk drives (e.g., DVDRAM), or other types of memory systems that retain data even after power is removed from the system. Non-volatile memory can also be random access memory.

[0101] Non-volatile memory can be a local device directly coupled to the rest of the data processing system. Alternatively, it can be a non-volatile memory located off-system, such as a network storage device coupled to the data processing system via a network interface, for example, a modem or Ethernet interface.

[0102] In this disclosure, some functions and operations are described as being executed by or caused by software code for the sake of simplicity. However, this expression is also used to indicate that the function is generated by a processor, such as a microprocessor, executing code / instructions.

[0103] Alternatively or in combination, the functions and operations described herein may be implemented using dedicated circuitry with or without software instructions, such as using application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). Embodiments may also be implemented using hard-wired circuitry systems without software instructions or in combination with software instructions. Therefore, the technology is neither limited to any specific combination of hardware circuitry systems and software, nor to any particular source of instructions executed by the data processing system.

[0104] While one embodiment may be implemented in a full-featured computer and computer system, various embodiments can be distributed as a variety of computing products and can be applied regardless of the specific type of machine or computer-readable medium actually used to implement the distribution.

[0105] At least some of the disclosed aspects may be embodied, at least in part, in software. That is, these techniques may be executed in a computer system or other data processing system in response to a processor (e.g., a microprocessor) executing a sequence of instructions contained in memory (e.g., ROM, volatile RAM, non-volatile memory, cache memory, or remote storage device).

[0106] The routines executed to implement the embodiments may be implemented as part of an operating system or a particular application, component, program, object, module, or a sequence of instructions referred to as a "computer program." A computer program typically contains one or more sets of instructions at various times in various memories and storage devices in a computer, and when read and executed by one or more processors in the computer, the sets of instructions cause the computer to perform necessary operations to perform elements involving various aspects.

[0107] Machine-readable media can be used to store software and data that enable a data processing system to perform various methods when executed by the system. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache memory. A portion of this software and / or data can be stored in any of these storage devices. Furthermore, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained at different times and in different communication sessions or in the same communication session from different centralized servers and / or peer-to-peer networks. All data and instructions can be obtained before the application is executed. Alternatively, portions of data and instructions can be obtained dynamically and as needed for execution. Therefore, it is not required that all data and instructions be on the machine-readable media at any given moment.

[0108] Examples of computer-readable media include, but are not limited to, media of non-transitory recordable and non-recordable types, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, disk storage media, optical storage media (e.g., optical disc read-only memory (CDROM), digital versatile disc (DVD), etc.), and others. Computer-readable media can store instructions.

[0109] Instructions can also be embodied in digital and analog communication links for use with electrical, optical, acoustic, or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, etc. However, propagated signals such as carrier waves, infrared signals, digital signals, etc., are not tangible machine-readable media and cannot be configured to store instructions.

[0110] In general, machine-readable media includes any entity that provides (i.e., stores and / or transmits) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device having one or more processors, etc.).

[0111] In various embodiments, hardwired circuitry systems can be used in combination with software instructions to implement the technology. Therefore, the technology is neither limited to any particular combination of hardware circuitry and software, nor to any particular source of instructions executed by the data processing system.

[0112] The above descriptions and figures are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are omitted to avoid obscuring the description. References to one or more embodiments in this disclosure do not necessarily refer to the same embodiment; and such references imply at least one.

[0113] In the foregoing description, this disclosure has been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made without departing from the broader spirit and scope set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative rather than restrictive sense.

Claims

1. An integrated circuit device, comprising: A computational logic set having processing units configured to perform computations on at least matrix operands; A random access memory coupled to the computational logic set and configured to store instructions executable by the computational logic set and to store matrices of an artificial neural network; as well as An interface that can be coupled to a memory controller external to the integrated circuit device and different from the set of computing logic, the interface being configured to facilitate the memory controller's access to the random access memory; In response to an instruction provided in the random access memory, the computational logic set is configured to autonomously execute the instructions without the aid of a processor to apply a first input stored in the random access memory to the artificial neural network, generate a first output from the artificial neural network, and store the first output in the random access memory.

2. The integrated circuit device according to claim 1, further comprising: An integrated circuit package configured to enclose the integrated circuit device; as well as A set of connectors configured to couple the interface to the memory controller located outside the integrated circuit package.

3. The integrated circuit device of claim 2, wherein the processing unit comprises a first unit configured to perform calculations on two matrix operands of an instruction.

4. The integrated circuit apparatus of claim 3, wherein the two matrix operands comprise a first matrix and a second matrix; the second matrix comprises a plurality of vectors; and the first unit comprises a plurality of second units configured to perform computations in parallel on the respective plurality of vectors of the first matrix and the second matrix.

5. The integrated circuit device of claim 4, wherein each of the plurality of second units comprises a plurality of third units configured to perform computations in parallel on a common vector and a plurality of vectors of the first matrix, respectively.

6. The integrated circuit device of claim 5, wherein each of the plurality of third units comprises a plurality of multiply-accumulate units configured to operate in parallel.

7. The integrated circuit device of claim 6, wherein the computational logic set is configured to autonomously execute the instructions when combining the first input and the matrix of the artificial neural network to produce the first output.

8. The integrated circuit device of claim 6, wherein the random access memory is configured to store a second input loaded through the interface during the execution of the instructions to operate on the first input; and after the first output is stored in the random access memory, the computational logic set is configured to execute the instructions to combine the second input and the matrix of the artificial neural network to produce a second output.

9. The integrated circuit device according to claim 8, further comprising: Local memory, which is coupled to the computational logic set; as well as The connection between the random access memory and the computational logic set; The local memory and the set of computing logic are configured as a deep learning accelerator on a first integrated circuit die, either a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The local memory is configured to cache matrix operands used by the first unit; and The random access memory is formed on one or more second integrated circuit dies; The connection is configured with bandwidth sufficient to load the matrix operands from the random access memory to the local memory during the time period in which the first unit performs operations on the two matrix operands.

10. The integrated circuit device of claim 9, wherein the bandwidth is sufficient to further store the result generated by the first unit from the local memory to the random access memory during the time period.

11. The integrated circuit device of claim 10, wherein the connection comprises a through-silicon via (TSV) between the first integrated circuit die and the one or more second integrated circuit dies.

12. The integrated circuit device of claim 6, wherein the random access memory includes a non-volatile memory configured to store the instructions and the matrix of the artificial neural network.

13. A method implemented in a computing device, comprising: The computing device accesses the random access memory of the computing device through an interface to a memory controller, the computing device having processing units configured to perform computations on at least matrix operands; Instructions executable by the processing unit are written into the random access memory through the interface. The matrix of the artificial neural network is written into the random access memory through the interface. The first input to the artificial neural network is written to the random access memory through the interface. The random access memory provides an instruction to cause the processing unit to begin executing the instructions, wherein the processing unit autonomously executes the instructions without the aid of a processor to combine the first input with the matrix of the artificial neural network to generate a first output from the artificial neural network and stores the first output in the random access memory; as well as The first output is read from the random access memory through the interface.

14. The method of claim 13, wherein the computing device is an integrated circuit device enclosed in an integrated circuit package; the integrated circuit device has a deep learning accelerator with the processing unit, control unit, and local memory; the processing unit includes at least a matrix-matrix unit configured to execute instructions having two matrix operands; the matrix-matrix unit includes a plurality of matrix-vector units configured to operate in parallel; each of the matrix-vector units includes a plurality of vector-vector units configured to operate in parallel; and each of the vector-vector units includes a plurality of multiply-accumulate units configured to operate in parallel.

15. The method of claim 14, further comprising: The description of the artificial neural network is converted into the instructions and the matrix.

16. The method of claim 15, further comprising: During the time period during which the deep learning accelerator executes the instructions to generate the first output from the first input according to the matrix of the artificial neural network, the second input to the artificial neural network is written to the random access memory through the interface; as well as After the first output is stored in the random access memory, an instruction is provided in the random access memory to cause the processing unit to begin executing the instruction to generate a second output from the second input.

17. The method of claim 16, wherein the reading of the first output from the random access memory is performed during a time period in which the deep learning accelerator executes the instructions to generate the second output from the second input according to the matrix of the artificial neural network.

18. A memory device comprising: Random access memory; Field-Programmable Gate Arrays (FPGAs) or Application-Specific Integrated Circuits (ASICs) have the following characteristics: A memory interface for accessing the random access memory; Control unit; as well as At least one processing unit configured to operate on two matrix operands of instructions executed in the FPGA or ASIC; as well as The connection between the random access memory and the computational logic set; as well as An interface to a memory controller, the interface being configured to facilitate the memory controller's access to the random access memory; In response to an instruction provided in the random access memory, the FPGA or ASIC is configured to autonomously execute instructions stored in the random access memory without the aid of a processor; and The FPGA or ASIC is configured to execute the instructions to generate a first output from the artificial neural network based on a first input stored in the random access memory and at least one matrix of the artificial neural network stored in the random access memory.

19. The memory device of claim 18, further comprising: An integrated circuit package that encloses the memory device; The FPGA or ASIC is formed on the first integrated circuit die.

20. The memory device of claim 19, wherein the random access memory comprises non-volatile memory formed on a second integrated circuit die; and the connection comprises a through-silicon via (TSV) between the first integrated circuit die and the second integrated circuit die.

Images