Discovering hardware characteristics of deep learning accelerators via compiler for optimization

Abstract

Systems, apparatuses, and methods related to deep learning accelerators and memory are described. For example, an integrated circuit apparatus can be configured to execute instructions having matrix operands and configured with random access memory. A computing apparatus running a compiler can interact with and / or probe the integrated circuit apparatus to identify hardware characteristics of the integrated circuit apparatus when performing matrix computations. The compiler can generate and optimize a compilation result from a description of an artificial neural network based at least in part on the hardware characteristics of the integrated circuit apparatus. The compilation result can include first data representing parameters of the artificial neural network and second data representing instructions executable by the integrated circuit apparatus to generate an output of the artificial neural network based on the first data and an input to the artificial neural network.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 092,033, filed November 6, 2020, entitled “Discovery of Hardware Characteristics of Deep Learning Accelerators for Optimization via Compiler,” the entire disclosure of which is hereby incorporated by reference. Technical Field

[0003] At least some of the embodiments disclosed herein generally relate to compilers, and more specifically (but not limited to) compilers for generating instructions that can be executed by accelerators for artificial neural networks (ANNs), such as ANNs configured via machine learning and / or deep learning. Background Technology

[0004] Artificial neural networks (ANNs) use a network of neurons to process inputs to the network and produce outputs from the network.

[0005] Deep learning has been used in many application areas, such as computer vision, speech / audio recognition, natural language processing, machine translation, bioinformatics, drug design, medical image processing, and games. Attached Figure Description

[0006] The embodiments are illustrated by way of example and are not limited to the figures in the accompanying drawings, wherein similar element symbols indicate similar elements.

[0007] Figure 1 An integrated circuit device with a deep learning accelerator and random access memory configured according to one embodiment is shown.

[0008] Figure 2 A processing unit configured to perform matrix-matrix operations according to one embodiment is shown.

[0009] Figure 3 A processing unit configured to perform matrix-vector operations according to one embodiment is shown.

[0010] Figure 4 A processing unit configured to perform vector-vector operations according to one embodiment is shown.

[0011] Figure 5 The illustration shows a deep learning accelerator and random access memory configured to autonomously apply input to a trained artificial neural network according to one embodiment.

[0012] Figure 6 This invention demonstrates a technique, according to one embodiment, for generating instructions executable by a deep learning accelerator to implement an artificial neural network.

[0013] Figure 7 and 8 This describes a technique, according to one embodiment, for mapping the compilation results of a general-purpose deep learning accelerator to instructions that can be executed by a specific deep learning accelerator to implement an artificial neural network.

[0014] Figure 9 This demonstrates another technique, according to one embodiment, for generating instructions executable by a deep learning accelerator to implement an artificial neural network.

[0015] Figure 10 An integrated circuit device having a deep learning accelerator with configurable hardware capabilities and random access memory is shown according to one embodiment.

[0016] Figure 11 This describes different hardware configurations of the processing unit of a deep learning accelerator, which can be configured via options stored in registers according to one embodiment.

[0017] Figure 12 This describes a technique, according to one embodiment, for generating instructions executable by a deep learning accelerator with optimized hardware configuration to implement an artificial neural network.

[0018] Figure 13 This paper demonstrates a technique for discovering hardware characteristics of deep learning accelerators according to one embodiment.

[0019] Figure 14 This describes a technique, according to one embodiment, for generating instructions that can be executed by a deep learning accelerator and optimized according to the hardware characteristics of the deep learning accelerator.

[0020] Figure 15 This paper demonstrates a method for compiling instructions to implement artificial neural networks on a deep learning accelerator based on hardware features of a deep learning accelerator, according to one embodiment.

[0021] Figure 16 A block diagram illustrating an example computer system in which embodiments of the present disclosure may be operated. Detailed Implementation

[0022] At least some embodiments disclosed herein provide an integrated circuit for implementing artificial neural network (ANN) computations with reduced power consumption and computation time. This integrated circuit device is programmable. A compiler can be used to generate instructions executable in the integrated circuit device from a description of the ANN. When executed in the device, the instructions cause the integrated circuit device to perform computations of the ANN. The compiler can discover hardware characteristics (e.g., capabilities and behavior) of a deep learning accelerator (DLA) and use the discovered hardware characteristics to optimize the generated instructions to implement the ANN computations on the DLA. For example, the integrated circuit device may include a deep learning accelerator (DLA) and random access memory. The random access memory is configured to store parameters of the ANN and instructions with matrix operations. The instructions stored in the random access memory can be executed by the deep learning accelerator (DLA) to perform matrix computations based on the ANN.

[0023] For example, a compiler can automatically train the hardware of a deep learning accelerator (DLA) to discover its characteristics, options, behavior, performance, latency, and / or limitations. For instance, the compiler can load a test instruction set into random access memory, cause the DLA to execute the instructions, examine the execution results, and / or measure the performance of instruction execution to discover the DLA's capabilities, performance, and / or behavior. Based on the discovered characteristics of the DLA across different hardware platforms, the compiler automatically customizes and / or optimizes the implementation of the same artificial neural network (ANN) on different hardware platforms.

[0024] For example, each neuron in a network receives a set of inputs. Some inputs to a neuron can be the output of a specific neuron in the network; and some inputs to a neuron can be the inputs to the neural network. The input / output relationship between neurons in a network represents the neuronal connectivity in the network.

[0025] For example, each neuron can have its own biases for its inputs, activation functions, and a set of synaptic weights. Activation functions can take the form of step functions, linear functions, log-sigmoid functions, etc. Different neurons in the network can have different activation functions.

[0026] For example, each neuron can produce a weighted sum of its input and its bias and then produce an output, which is a function calculated using the neuron's activation function as a weighted sum.

[0027] Generally, the relationship between the inputs and outputs of an ANN is defined by the ANN model, which contains data representing the connectivity of neurons in the network, as well as the biases, activation functions, and synaptic weights of each neuron. Based on a given ANN model, a computing device can be configured to compute the network's outputs from a given set of inputs to the network.

[0028] For example, the input to the ANN network can be generated based on camera input; and the output from the ANN network can be used for the identification of items, such as events or objects.

[0029] Generally, ANNs can be trained using supervised methods, where the parameters of the ANN are tuned to minimize or reduce the error between a known output associated with or derived from a given input and the output obtained by applying the input to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.

[0030] Alternatively, or in combination, ANNs can be trained using unsupervised methods, where the exact output derived from a given set of inputs is unknown until training is complete. ANNs can be trained to classify items into multiple categories or data points into clusters.

[0031] A variety of training algorithms can be used for complex machine learning / training paradigms.

[0032] Deep learning uses multi-layered machine learning to progressively extract features from input data. For example, lower layers can be configured to identify edges in an image; and higher layers can be configured to identify captured items in the image, such as faces, objects, events, etc., based on the edges detected by the lower layers. Deep learning can be implemented via artificial neural networks (ANNs), such as deep neural networks, deep belief networks, recurrent neural networks, and / or convolutional neural networks.

[0033] A typical deep learning accelerator (DLA) may contain a set of programmable hardware computing logic that is specialized and / or optimized to perform parallel vector and / or matrix computations, including but not limited to vector and / or matrix multiplication and accumulation.

[0034] In addition, deep learning accelerators may contain one or more arithmetic logic units (ALUs) to perform arithmetic and bitwise operations on integer binary numbers.

[0035] Deep learning accelerators can be programmed via a set of instructions to perform computations in artificial neural networks (ANNs).

[0036] The granularity of deep learning accelerators performing operations on vectors and matrices corresponds to the largest unit of vector / matrix that can be operated on during the execution of a single instruction by the deep learning accelerator. During instruction execution for predefined operations on vector / matrix operands, deep learning accelerators can perform operations on the elements of vector / matrix operands in parallel to reduce execution time and / or energy consumption associated with memory / data access. Operations on vector / matrix operands at the deep learning accelerator granularity can be used as building blocks to perform computations on larger vectors / matrices.

[0037] The implementation of typical / practical artificial neural networks involves vector / matrix operations with a granularity larger than that of deep learning accelerators. To implement this artificial neural network using a deep learning accelerator, computations involving large vector / matrix operations can be decomposed into computations of vector / matrix operations at the deep learning accelerator granularity. Deep learning accelerators can be programmed via instructions to perform computations involving large vector / matrix operations. For example, the ability of a deep learning accelerator to manipulate atomic computations of vectors and matrices at the deep learning accelerator granularity in response to instructions can be programmed to perform computations within an artificial neural network.

[0038] In some implementations, deep learning accelerators lack some of the logical operational capabilities of a typical central processing unit (CPU). However, deep learning accelerators can be configured with sufficient logic units to process the input data provided to the artificial neural network and to produce the output of the artificial neural network based on a set of instructions generated for the deep learning accelerator. Therefore, deep learning accelerators can perform computations of the artificial neural network with little or no assistance from a central processing unit (CPU) or another processor. Optionally, a conventional general-purpose processor can also be configured as part of a deep learning accelerator to perform operations that cannot be efficiently implemented using the vector / matrix processing unit of the deep learning accelerator and / or cannot be performed by the vector / matrix processing unit of the deep learning accelerator.

[0039] Typical artificial neural networks can be described / specified in a standard format (e.g., Open Neural Network Exchange (ONNX)). A compiler can be used to translate the description of an artificial neural network into a set of instructions for deep learning accelerators to perform computations on the artificial neural network. The compiler can optimize this set of instructions to improve the performance of deep learning accelerators when implementing artificial neural networks.

[0040] Deep learning accelerators may have local memory, such as registers, buffers, and / or caches, configured to store vector / matrix operands and the results of vector / matrix operations. Intermediate results in registers can be pipelined / shifted within the deep learning accelerator as operands for subsequent vector / matrix operations, reducing the time and energy consumption of accessing memory / data and thus accelerating the implementation of typical vector / matrix operations in typical artificial neural networks. The capacity of registers, buffers, and / or caches in a deep learning accelerator is often insufficient to hold the entire dataset used for computations implementing typical artificial neural networks. Therefore, random access memory coupled to the deep learning accelerator is configured to provide improved data storage capabilities for implementing typical artificial neural networks. For example, the deep learning accelerator loads data and instructions from random access memory and stores the results back into random access memory.

[0041] The communication bandwidth between the deep learning accelerator and random access memory (RAM) is configured to optimize or maximize the utilization of the deep learning accelerator's computational power. For example, high communication bandwidth can be provided between the deep learning accelerator and RAM, allowing vector / matrix operands to be loaded from RAM into the deep learning accelerator and the results stored back in RAM within a timeframe approximately equal to the time it takes for the deep learning accelerator to perform computations on the vector / matrix operands. The granularity of the deep learning accelerator can be configured to increase the ratio between the computational load performed by the deep learning accelerator and the size of the vector / matrix operands, thereby reducing data access traffic between the deep learning accelerator and RAM. This reduces the demand for communication bandwidth between the deep learning accelerator and RAM. Therefore, bottlenecks in data / memory access can be reduced or eliminated.

[0042] Optionally, the compiler can be configured to support different hardware platforms for deep learning accelerators. Specifically, the compiler can generate different instruction sets for different deep learning accelerators based on the same description of the artificial neural network. For example, deep learning accelerators can be implemented using different technologies, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). For example, deep learning accelerators can have different hardware capabilities when performing matrix operations, with different numbers of parallel processing units operable to concurrently perform matrix operations, and / or different computational granularities, where the processing units can have different capabilities to process matrices of different sizes when executing instructions with matrix operands. The compiler can initially apply general, platform-independent optimizations to the description of the artificial neural network to generate a general computational model based on the common characteristics of computations implemented using different deep learning accelerators. The compiler then maps the compiled result of the general computational model to different hardware platforms / implementations of the deep learning accelerator. Optionally, the compiler can further optimize the compiled result for individual types of deep learning accelerators to reduce power consumption and / or computation time.

[0043] In some implementations, deep learning accelerators (DLAs) are designed with multiple configurable hardware options. Different hardware options may be optimal for different scenarios of artificial neural network computation. During the compilation and optimization of the artificial neural network, the compiler is configured to optimize the instructions generated for execution by the deep learning accelerator. Compiler optimization may include the selection of hardware options to improve the overall performance of the artificial neural network (ANN) implementation within the deep learning accelerator. Therefore, the compiler can optimize and / or customize the circuit configuration of the deep learning accelerator itself when implementing a specific artificial neural network.

[0044] Figure 1 An integrated circuit device (101) with a deep learning accelerator (103) and random access memory (105) configured according to one embodiment is shown.

[0045] Figure 1 The deep learning accelerator (103) includes a processing unit (111), a control unit (113), and local memory (115). When vector and matrix operands are in local memory (115), the control unit (113) can use the processing unit (111) to perform vector and matrix operations according to instructions. In addition, the control unit (113) can load instructions and operands from random access memory (105) through memory interface (117) and high-speed / bandwidth connection (119).

[0046] The integrated circuit device (101) is configured to be enclosed in an integrated circuit package having pins or contacts for a memory controller interface (107).

[0047] The memory controller interface (107) is configured to support standard memory access protocols, so that the integrated circuit device (101) is presented to a typical memory controller in the same way as a conventional random access memory device without a deep learning accelerator (103). For example, a memory controller external to the integrated circuit device (101) can access the random access memory (105) in the integrated circuit device (101) using standard memory access protocols through the memory controller interface (107).

[0048] The integrated circuit device (101) is connected to a deep learning accelerator (103) via a high-bandwidth connection (119) configured to surround a random access memory (105) within the integrated circuit device (101). The bandwidth of the connection (119) is higher than the bandwidth of the connection (109) between the random access memory (105) and the memory controller interface (107).

[0049] In one embodiment, both the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via the same set of buses or wires. Therefore, the bandwidth for accessing the random access memory (105) is shared between the memory interface (117) and the memory controller interface (107). Optionally, the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via separate sets of buses or wires. Optionally, the random access memory (105) may comprise multiple segments that can be accessed concurrently via a connection (119). For example, while the memory interface (117) is accessing a segment of the random access memory (105), the memory controller interface (107) may concurrently access another segment of the random access memory (105). For example, different segments can be configured on different planes / memory banks of different integrated circuit dies and / or memory cells; and different segments can be accessed in parallel to increase the throughput of accessing the random access memory (105). For example, the memory controller interface (107) is configured to access a data cell of a predetermined size at a time; and the memory interface (117) is configured to access multiple data cells at a time, each data cell having the same predetermined size.

[0050] In one embodiment, the random access memory (105) and the integrated circuit device (101) are disposed on different integrated circuit dies disposed within the same integrated circuit package. Furthermore, the random access memory (105) may be disposed on one or more integrated circuit dies that allow concurrent parallel access to multiple data elements.

[0051] In some implementations, the number of data elements of a vector or matrix that can be accessed in parallel via connection (119) corresponds to the granularity of the deep learning accelerator for vector or matrix operations. For example, when the processing unit (111) can perform operations on several vector / matrix elements in parallel, connection (119) is configured to load or store the same number or a multiple thereof of elements in parallel via connection (119).

[0052] Optionally, the data access speed of the connection (119) can be configured based on the processing speed of the deep learning accelerator (103). For example, after a certain amount of data and instructions have been loaded into the local memory (115), the control unit (113) can use the processing unit (111) to execute instructions to perform operations on the data to produce output. During the processing time used to produce output, the access bandwidth of the connection (119) allows the same amount of data and instructions to be loaded into the local memory (115) for the next operation, and allows the same amount of output to be stored back into the random access memory (105). For example, while the control unit (113) is using a portion of the local memory (115) to process data and produce output, the memory interface (117) can unload the output of the previous operation from another portion of the local memory (115) into the random access memory (105) and load the operand data and instructions into said other portion. Therefore, the utilization and performance of the deep learning accelerator are not limited or reduced by the bandwidth of the connection (119).

[0053] Random access memory (105) can be used to store model data of the artificial neural network and buffer input data of the artificial neural network. The model data does not change frequently. The model data may contain outputs generated by the compiler for the deep learning accelerator to implement the artificial neural network. The model data typically contains matrices used in the description of the artificial neural network and instructions generated for the deep learning accelerator (103) to perform vector / matrix operations of the artificial neural network at the granularity of the deep learning accelerator (103). The instructions operate not only on the vector / matrix operations of the artificial neural network but also on the input data of the artificial neural network.

[0054] In one embodiment, when input data is loaded or updated into random access memory (105), the control unit (113) of the deep learning accelerator (103) can automatically execute instructions of the artificial neural network to produce the output of the artificial neural network. The output is stored in a predetermined area in the random access memory (105). The deep learning accelerator (103) can execute instructions without the assistance of a central processing unit (CPU). Therefore, communication for coordination between the deep learning accelerator (103) and a processor (e.g., a central processing unit (CPU)) outside the integrated circuit device (101) can be reduced or eliminated.

[0055] Optionally, the logic circuitry of the deep learning accelerator (103) may be implemented using complementary metal-oxide-semiconductor (CMOS). For example, the logic circuitry of the deep learning accelerator (103), including the processing unit (111) and the control unit (113), may be implemented using under-array CMOS (CUA) technology of the memory cells of the random access memory (105). Alternatively, the logic circuitry of the deep learning accelerator (103) may be implemented using CMOS technology of the memory cells of the random access memory (105).

[0056] In some implementations, the deep learning accelerator (103) and random access memory (105) can be implemented on separate integrated circuit dies and connected using through-silicon vias (TSVs) to increase the data bandwidth between the deep learning accelerator (103) and the random access memory (105). For example, the deep learning accelerator (103) can be formed on an integrated circuit die of a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0057] Alternatively, the deep learning accelerator (103) and random access memory (105) can be configured in a separate integrated circuit package and connected via multiple point-to-point connections on a printed circuit board (PCB) for parallel communication, thereby increasing data transmission bandwidth.

[0058] Random access memory (105) can be volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Examples of non-volatile memory include flash memory, memory cells formed based on NAND and NOR logic gates, phase-change memory (PCM), magnetic RAM, resistive random access memory, cross-point memory devices, and memory devices. Cross-point memory devices can use transistorless memory elements, each of which has memory cells and selectors stacked together in a column. The column of memory elements is connected via two layers of wires extending in a vertical direction, one layer of wires extending in one direction in a layer positioned above the column of memory elements, and the other layer of wires extending in another direction and positioned below the column of memory elements. Each memory element can be individually selected at the intersection of a wire on each of the two layers. Cross-point memory devices are fast and non-volatile and can be used as a unified memory set for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0059] For example, non-volatile memory can be configured to implement at least a portion of random access memory (105). The non-volatile memory within random access memory (105) can be used to store model data of the artificial neural network. Therefore, after the integrated circuit device (101) is powered off and restarted, it is not necessary to reload the model data of the artificial neural network into the integrated circuit device (101). Furthermore, the non-volatile memory can be programmable / rewritable. Therefore, the model data of the artificial neural network in the integrated circuit device (101) can be updated or replaced to implement an updated artificial neural network or another artificial neural network.

[0060] The processing unit (111) of the deep learning accelerator (103) may include vector-vector units, matrix-vector units, and / or matrix-matrix units. The following is in conjunction with... Figures 2 to 4 This section discusses instances of units configured to perform vector-vector operations, matrix-vector operations, and matrix-matrix operations.

[0061] Figure 2 A processing unit configured to perform matrix-matrix operations according to one embodiment is shown. For example, Figure 2 The matrix-matrix unit (121) can be used as Figure 1 One of the processing units (111) of the deep learning accelerator (103).

[0062] exist Figure 2 In this configuration, the matrix-matrix unit (121) includes multiple kernel buffers (131 to 133) and multiple mapped memory banks (151 to 153). Each mapped memory bank (151 to 153) stores a vector of a matrix operand, which has multiple vectors stored in the mapped memory bank (151 to 153); and each kernel buffer (131 to 133) stores a vector of another matrix operand, which has multiple vectors stored in the kernel buffer (131 to 133). The matrix-matrix unit (121) is configured to perform multiplication and accumulation operations on the elements of the two matrix operands using multiple matrix-vector units (141 to 143) that operate in parallel.

[0063] A crossbar switch (123) connects the mapped memory (151 to 153) to the matrix-vector units (141 to 143). The same matrix operands stored in the mapped memory (151 to 153) are provided to each matrix-vector unit (141 to 143) via the crossbar switch (123); and the matrix-vector units (141 to 143) receive data elements from the mapped memory (151 to 153) in parallel. Each kernel buffer (131 to 133) is connected to a corresponding matrix-vector unit (141 to 143) and provides vector operands to the corresponding matrix-vector unit. The matrix-vector units (141 to 143) operate concurrently to compute the operation of multiplying the same matrix operands stored in the mapped memory (151 to 153) by the corresponding vector stored in the kernel buffer (131 to 133). For example, the matrix-vector unit (141) performs multiplication on matrix operands stored in the mapped memory (151 to 153) and vector operands stored in the kernel buffer (131), while the matrix-vector unit (143) concurrently performs multiplication on matrix operands stored in the mapped memory (151 to 153) and vector operands stored in the kernel buffer (133).

[0064] Figure 2 Each of the matrix-vector units (141 to 143) in the matrix can Figure 3 Implement it in the manner described herein.

[0065] Figure 3 A processing unit configured to perform matrix-vector operations according to one embodiment is shown. For example, Figure 3 The matrix-vector unit (141) can be used as Figure 2 Any matrix-vector unit in the matrix-matrix unit (121).

[0066] exist Figure 3In the diagram, each mapped memory (151 to 153) is configured in a manner similar to... Figure 2 A vector of matrix operands is stored in a mapped memory (151 to 153), wherein the matrix operands have multiple vectors stored in the mapped memory (151 to 153). Figure 3 The crossbar switch (123) in the middle provides vectors from the mapped memory (151) to vector-vector units (161 to 163). The same vectors stored in the kernel buffer (131) are provided to vector-vector units (161 to 163).

[0067] Vector-vector units (161 to 163) concurrently operate to compute the operation of multiplying the corresponding vector operands stored in the mapped memory (151 to 153) by the same vector operands stored in the kernel buffer (131). For example, vector-vector unit (161) performs multiplication on the vector operands stored in the mapped memory (151) and the vector operands stored in the kernel buffer (131), while vector-vector unit (163) concurrently performs multiplication on the vector operands stored in the mapped memory (153) and the vector operands stored in the kernel buffer (131).

[0068] when Figure 3 The matrix-vector unit (141) in Figure 2 When implemented in the matrix-matrix unit (121), the matrix-vector unit (141) may use the mapping memory (151 to 153), crossbar switch (123) and kernel buffer (131) of the matrix-matrix unit (121).

[0069] Figure 3 Each of the vector-vector units (161 to 163) in the vector array can Figure 4 Implement it in the manner described herein.

[0070] Figure 4 A processing unit configured to perform vector-vector operations according to one embodiment is shown. For example, Figure 4 The vector-vector unit (161) can be used as Figure 3 Any vector-vector unit in the matrix-vector unit (141).

[0071] exist Figure 4 In this context, the vector-vector unit (161) has multiple multiplication-accumulation units (171 to 173). Each multiplication-accumulation unit (e.g., 173) can receive two numbers as operands, perform multiplication of the two numbers, and add the result of the multiplication to the sum maintained in the multiplication-accumulation unit.

[0072] Each vector buffer (181 and 183) stores a sequence of numbers. A pair of numbers (each from one of the vector buffers (181 and 183)) can be provided as input to each multiply-accumulate unit (171 to 173). The multiply-accumulate units (171 to 173) can receive multiple pairs of numbers from the vector buffers (181 and 183) in parallel and perform multiply-accumulate (MAC) operations in parallel. The outputs from the multiply-accumulate units (171 to 173) are stored in a shift register (175); and the accumulator (177) calculates the sum of the results in the shift register (175).

[0073] when Figure 4 The vector-vector unit (161) in Figure 3 When implemented in the matrix-vector unit (141), the vector-vector unit (161) may use a mapped memory (e.g., 151 or 153) as a vector buffer (181) and the kernel buffer (131) of the matrix-vector unit (141) as another vector buffer (183).

[0074] Vector buffers (181 and 183) may have the same length to store the same number / count of data elements. The length may be equal to or a multiple of the count of the multiplication-accumulation units (171 to 173) in the vector-vector unit (161). When the length of the vector buffers (181 and 183) is a multiple of the count of the multiplication-accumulation units (171 to 173), multiple pairs of inputs (equal to the count of the multiplication-accumulation units (171 to 173)) may be provided from the vector buffers (181 and 183) as inputs to the multiplication-accumulation units (171 to 173) in each iteration; and the vector buffers (181 and 183) feed their elements into the multiplication-accumulation units (171 to 173) through multiple iterations.

[0075] In one embodiment, the communication bandwidth of the connection (119) between the deep learning accelerator (103) and the random access memory (105) is sufficient to enable the matrix-matrix unit (121) to use a portion of the random access memory (105) as a mapping memory (151 to 153) and a kernel buffer (131 to 133).

[0076] In another embodiment, a mapped memory (151 to 153) and a kernel buffer (131 to 133) are implemented in a portion of the local memory (115) of the deep learning accelerator (103). The communication bandwidth of the connection (119) between the deep learning accelerator (103) and the random access memory (105) is sufficient to load the matrix operands of the next operation cycle of the matrix-matrix unit (121) into another portion of the local memory (115), while the matrix-matrix unit (121) is performing computations in the current operation cycle using the mapped memory (151 to 153) and the kernel buffer (131 to 133) implemented in different portions of the local memory (115) of the deep learning accelerator (103).

[0077] Figure 5 The illustration shows a deep learning accelerator and random access memory configured to autonomously apply input to a trained artificial neural network according to one embodiment.

[0078] Artificial neural networks (201) trained by machine learning (e.g., deep learning) can be described in a standard format (e.g., Open Neural Network Exchange (ONNX)). A standard format description of a trained artificial neural network (201) identifies the properties of the artificial neurons and their connectivity.

[0079] exist Figure 5 In this process, the deep learning accelerator compiler (203) transforms the trained artificial neural network (201) by generating instructions (205) for the deep learning accelerator (103) and a matrix (207) corresponding to the properties and connectivity of the artificial neurons. The instructions (205) and the matrix (207) generated by the DLA compiler (203) from the trained artificial neural network (201) can be stored in the random access memory (105) of the deep learning accelerator (103).

[0080] For example, random access memory (105) and deep learning accelerator (103) can be connected via high-bandwidth connection (119) as follows: Figure 1 The integrated circuit device (101) is connected in a manner based on instructions (205) and a matrix (207). Figure 5 Autonomous computing can Figure 1 The integrated circuit device (101) is implemented. Alternatively, the random access memory (105) and the deep learning accelerator (103) may be configured on a printed circuit board having multiple point-to-point serial buses extending in parallel for connection (119).

[0081] exist Figure 5In the process, after the result of the DLA compiler (203) is stored in the random access memory (105), the application of the trained artificial neural network (201) can be triggered by the presence of an input (211) in the random access memory (105) or another indication provided in the random access memory (105) to process the input (211) to the trained artificial neural network (201) to produce the corresponding output (213) of the trained artificial neural network (201).

[0082] In response, the deep learning accelerator (103) executes instructions (205) to combine the input (211) and the matrix (207). The matrix (207) may contain a kernel matrix to be loaded into a kernel buffer (131 to 133) and a mapping matrix to be loaded into a mapping memory (151 to 153). The execution of instructions (205) may involve generating a mapping matrix for a mapping memory (151 to 153) of one or more matrix-matrix units (e.g., 121) of the deep learning accelerator (103).

[0083] In some embodiments, the input to the artificial neural network (201) is in the form of an initial mapping matrix. A portion of the initial mapping matrix may be retrieved from random access memory (105) as matrix operands stored in mapping storage blocks (151 to 153) of matrix-matrix units (121). Alternatively, the DLA instructions (205) may also include instructions for the deep learning accelerator (103) to generate the initial mapping matrix from the input (211).

[0084] According to the DLA instruction (205), the deep learning accelerator (103) loads matrix operands into the kernel buffer (131 to 133) and mapping memory (151 to 153) of its matrix-matrix unit (121). The matrix-matrix unit (121) performs matrix computations on the matrix operands. For example, the DLA instruction (205) decomposes the matrix computations of the trained artificial neural network (201) according to the computational granularity of the deep learning accelerator (103) (e.g., the size / dimensionality of the matrix loaded as a matrix operand in the matrix-matrix unit (121)) and applies the input feature maps to the kernel of the artificial neuron layer to produce outputs as inputs to the next layer of artificial neurons.

[0085] When the computation of the trained artificial neural network (201) executed according to the instruction (205) is completed, the deep learning accelerator (103) stores the output (213) of the artificial neural network (201) at a predefined location in the random access memory (105) or at a location specified in the instruction provided in the random access memory (105) to trigger the computation.

[0086] when Figure 5 The technology in Figure 1When implemented in the integrated circuit device (101), an external device connected to the memory controller interface (107) can write the input (211) into the random access memory (105) and trigger the deep learning accelerator (103) to apply the input (211) to the autonomous computation of the trained artificial neural network (201). After a period of time, the output (213) becomes available in the random access memory (105); and the external device can read the output (213) via the memory controller interface (107) of the integrated circuit device (101).

[0087] For example, a predefined location in random access memory (105) can be configured to store an indication that triggers the autonomous execution of instructions (205) by the deep learning accelerator (103). The indication may optionally include the location of the input (211) within random access memory (105). Therefore, during the autonomous execution of instructions (205) to process the input (211), an external device can retrieve outputs generated during a previous run of the instruction (205) and / or store another set of inputs for the next run of the instruction (205).

[0088] Optionally, another predefined location in the random access memory (105) may be configured to store an indication of the current running progress status of the instruction (205). Furthermore, the indication may include a prediction of the completion time of the current running of the instruction (205) (e.g., an estimate based on previous runs of the instruction (205)). Thus, an external device can check the completion status within an appropriate time window to retrieve the output (213).

[0089] In some embodiments, the random access memory (105) is configured to have sufficient capacity to store multiple sets of inputs (e.g., 211) and outputs (e.g., 213). Each set may be configured in a predetermined slot / region in the random access memory (105).

[0090] The deep learning accelerator (103) can autonomously execute instructions (205) to produce an output (213) from an input (211) based on a matrix (207) stored in random access memory (105) without the assistance of a processor or device located outside the integrated circuit device (101).

[0091] In one embodiment of the method, an interface (107) from a computing device (e.g., an integrated circuit device (101)) to a memory controller can be used to access the random access memory (105) of the computing device. The computing device may have a processing unit (e.g., 111) configured to perform calculations on at least matrix operands, such as matrix operands stored in mapped memory (151 to 153) and matrix operands stored in kernel buffers (131 to 133).

[0092] For example, a computing device implemented using an integrated circuit device (101) and / or other components may be enclosed within an integrated circuit package; and a set of connections may connect an interface (107) to a memory controller located outside the integrated circuit package.

[0093] Instructions (205) that can be executed by a processing unit (e.g., 111) can be written into random access memory (105) via an interface (107).

[0094] The matrix (207) of the artificial neural network (201) can be written into the random access memory (105) via the interface (107). The matrix (207) identifies the parameters, properties and / or state of the artificial neural network (201).

[0095] Optionally, at least a portion of the random access memory (105) is non-volatile and configured to store instructions (205) and matrices (207) of the artificial neural network (201).

[0096] The first input (211) of the artificial neural network can be written into the random access memory (105) through the interface (107).

[0097] An instruction is provided in the random access memory (105) to cause the processing unit (111) to begin executing instructions (205). In response to the instruction, the processing unit (111) executes instructions to combine the first input (211) of the artificial neural network (201) with the matrix (207) to generate a first output (213) from the artificial neural network (201) and stores the first output (213) in the random access memory (105).

[0098] For example, the indication may be the address of a first input (211) in random access memory (105); and the indication may be stored at a predetermined location in random access memory (105) such that execution of an instruction (205) for the input (211) identified by the address is initiated. Optionally, the indication may also include an address for storing an output (213).

[0099] The first output (213) can be read from the random access memory (105) via the interface (107).

[0100] For example, a computing device (e.g., an integrated circuit device (101)) may have a deep learning accelerator (103) formed on a first integrated circuit die and random access memory (105) formed on one or more second integrated circuit dies. The connection (119) between the first integrated circuit die and one or more second integrated circuit dies may include through-silicon vias (TSVs) to provide high bandwidth for memory access.

[0101] For example, the description of the artificial neural network (201) can be converted into instructions (205) and matrices (207) using a compiler (203). The combination of instructions (205) and matrices (207) stored in random access memory (105) and deep learning accelerator (103) provides an autonomous implementation of the artificial neural network (201) that can automatically convert the input (211) to the artificial neural network (201) into its output (213).

[0102] For example, during the time period in which the deep learning accelerator (103) executes instructions (205) according to the matrix (207) of the artificial neural network (201) to produce a first output (213) from a first input (211), a second input to the artificial neural network (201) can be written into random access memory (105) at an alternative location via interface (107). After the first output (213) is stored in random access memory (105), an instruction can be provided in the random access memory to cause the deep learning accelerator (103) to resume execution of instructions and produce a second output from the second input.

[0103] During the time period in which the deep learning accelerator (103) executes instructions (205) according to the matrix (207) of the artificial neural network (201) to generate a second output from a second input, the first output (213) can be read from the random access memory (105) via the interface (107); and another input can be written into the random access memory to replace the first input (211), or written to a different location. The process can be repeated for a series of inputs.

[0104] The deep learning accelerator (103) may include at least one matrix-matrix unit (121) capable of executing instructions on two matrix operands. The two matrix operands may be a first matrix and a second matrix. Each of the two matrices has multiple vectors. The matrix-matrix unit (121) may include multiple matrix-vector units (141 to 143) configured for parallel operation. Each of the matrix-vector units (141 to 143) is configured to operate on the first matrix and a vector from the second matrix in parallel with other matrix-vector units. Furthermore, each of the matrix-vector units (141 to 143) may have multiple vector-vector units (161 to 163) configured for parallel operation. Each of the vector-vector units (161 to 163) is configured to operate on a vector from the first matrix and a common vector operand of the corresponding matrix-vector unit in parallel with other vector-vector units. In addition, each of the vector-vector units (161 to 163) may have multiple multiplication-accumulation units (171 to 173) configured to operate in parallel.

[0105] In addition to the processing unit (111), the deep learning accelerator (103) may have local memory (115) and a control unit (113). The control unit (113) may load instructions (205) and matrix operands (e.g., some of the matrices (207)) from random access memory (105) for execution by the processing unit (111). The local memory may cache the matrix operands used by the matrix-matrix unit. The connection (119) may be configured with sufficient bandwidth to load a set of matrix operands from random access memory (105) into local memory (115) during a time period in which the matrix-matrix unit performs operations on two other matrix operands. Furthermore, during said time period, the bandwidth is sufficient to store the results produced by the matrix-matrix unit (121) in the previous instruction execution from local memory (115) into random access memory (105).

[0106] At least some of the embodiments disclosed herein provide a compiler that can translate the same description of an artificial neural network into different instruction sets that can be executed on different hardware platforms of a deep learning accelerator.

[0107] Deep learning accelerators can be implemented using different integrated circuit technologies, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Furthermore, deep learning accelerators can have different hardware capabilities when performing matrix operations.

[0108] For example, different hardware implementations of deep learning accelerators may have different numbers of parallel processing units that can operate to perform matrix operations concurrently.

[0109] For example, different hardware implementations of deep learning accelerators can have different matrix computation granularities. Instructions can be used to perform predefined matrix operations on matrix operands. However, the dimension of the matrix operands in the instructions can vary from deep learning accelerator to deep learning accelerator.

[0110] In one embodiment, the compiler is configured to initially perform platform-independent compilation and optimization for a general-purpose deep learning accelerator. The hardware capabilities of the general-purpose deep learning accelerator are predefined to capture common characteristics of several different deep learning accelerators. The compilation results of the general-purpose deep learning accelerator can be mapped to the compilation results of different deep learning accelerators. Thus, the same description of an artificial neural network can be compiled into different instruction sets that can be executed on different deep learning accelerators that use different integrated circuit technologies (e.g., FPGA or ASIC) and / or are implemented with different granularities and parallel execution capabilities. Optionally, the compiler can further optimize the compilation results for individual types of deep learning accelerators to further reduce power consumption and / or computation time.

[0111] Figure 6 This invention demonstrates a technique, according to one embodiment, for generating instructions executable by a deep learning accelerator to implement an artificial neural network.

[0112] exist Figure 6 In the description (221), the ANN description identifies the parameters of the artificial neural network (201), including the behavioral model of the artificial neurons and the connectivity of the artificial neurons in the network. For example, the parameters may include the identification of the activation function, bias, and / or state of the artificial neurons. For example, the parameters may include the synaptic weights of the connections between the artificial neurons. The description (221) in a standard format (e.g., Open Neural Network Exchange (ONNX)) may be provided as input to the DLA compiler (203).

[0113] The DLA compiler (203) performs compilation and optimization (223) according to the general DLA specification (225). The general DLA specification (225) identifies the computational capabilities of general deep learning accelerators.

[0114] For example, a general-purpose deep learning accelerator may have the common hardware features of many deep learning accelerators, which can be implemented using different technologies, with different granularities and different capacities.

[0115] For example, a general-purpose deep learning accelerator can be built as a virtual deep learning accelerator to be implemented on a specific hardware platform for deep learning accelerators.

[0116] For example, a general-purpose deep learning accelerator can be described as a platform-independent feature of a class of deep learning accelerators that can be implemented via ASIC, FPGA, or another technology.

[0117] The DLA compiler (203) produces a general result (227) by compiling and optimizing (223) a general deep learning accelerator. For example, the general result (227) may contain instructions for matrix computations on a general or virtual deep learning accelerator that conforms to the general DLA specification (225) to implement an artificial neural network (201).

[0118] The DLA compiler (203) can further perform a DLA mapping (233), which maps a general result (227) to a compiler output (237) specific to the hardware platform of the deep learning accelerator. A specific DLA specification (235) identifies the hardware capabilities of the specific hardware platform of the deep learning accelerator. The compiler output (237) contains DLA instructions (205) executable on a deep learning accelerator (103) conforming to the specific DLA specification (235). The compiler output (237) further contains a DLA matrix (207) representing the parameters of the artificial neural network (201).

[0119] Optionally, some aspects of a general-purpose deep learning accelerator can be parameterized, such as the number of processing units of a predetermined type that can be operated to process data in parallel, the processing granularity of the processing units, etc. Thus, such aspects of a general-purpose deep learning accelerator can be configured for compilation and optimization (223) to produce a general result (227) that matches the optimization result of a specific DLA specification (235) through a DLA mapping (233).

[0120] The DLA compiler (203) can map the general results (227) compiled for the general deep learning accelerator to the platform-specific compiler output (237) of the deep learning accelerator by implementing the instructions and / or routines of the general deep learning accelerator using platform-specific instructions and routines.

[0121] Figure 7 and 8 This describes a technique, according to one embodiment, for mapping the compilation results of a general-purpose deep learning accelerator to instructions that can be executed by a specific deep learning accelerator to implement an artificial neural network.

[0122] Figure 7 This describes a technique for using DLA routines (e.g., 243) to map instructions of a general-purpose deep learning accelerator to DLA instructions (205) that can be executed on a hardware platform specified or identified by a particular DLA specification (235).

[0123] For example, a generic DLA instruction (241) can be implemented using a DLA routine (243) that can be executed on a specific hardware platform. The use of the generic DLA instruction (241) in the compiled generic result (227) can be replaced by using a DLA routine (243) configured according to the specific DLA specification (235) for a specific hardware platform.

[0124] For example, DLA routines (243) can be pre-optimized to implement general DLA instructions (241) on hardware platforms with specific DLA specifications (235).

[0125] exist Figure 8 In this context, a general routine (245) implemented using instructions according to a general DLA specification (225) is mapped to a DLA routine (247) implemented using instructions according to a specific DLA specification (225). The DLA routine (247) can be pre-optimized to improve the performance of the overall task performed by the routine, such that the performance of the DLA routine (247) is better than replacing the corresponding general DLA instruction (e.g., 241) in the general routine (245) with the corresponding DLA routine (e.g., 243).

[0126] Generally, different routines or instruction combinations in the general result (227) may contribute differently to the performance of the compiled general result (227) when implementing the computation of the artificial neural network (201). Routines or instruction combinations with a larger share of computational workload can be mapped to optimized DLA routines (e.g., 247) to improve the performance of the compiler output (237).

[0127] Optionally, after the DLA mapping (233), the DLA compiler (203) may further perform optimizations to improve the performance of the compiler output (237), such as... Figure 9 As explained.

[0128] Figure 9 This demonstrates another technique, according to one embodiment, for generating instructions executable by a deep learning accelerator to implement an artificial neural network.

[0129] exist Figure 9 In this context, the DLA compiler (203) can be based on the ANN description (221) and the general DLA specification (225) in a manner similar to... Figure 6 The initial compilation and optimization (223) of the artificial neural network (201) are performed in a manner that allows for further processing. Furthermore, the DLA compiler (203) can perform DLA mapping (233) according to a specific DLA specification (235) to convert the general compilation result (227) into a mapping result (229) for implementation. This can be used... Figure 7 and 8 The technology is used to perform DLA mapping (233).

[0130] Following the DLA mapping (233), the DLA compiler (203) may further perform optimizations (231) on the compiled mapped result (229) to produce compiler output (237). For example, the DLA compiler (203) may transform the mapped result (229) to reduce the energy consumption and / or computation time of implementing the ANN description (221) on a platform identified by a specific DLA specification (235).

[0131] In one embodiment of the method, the compiler translates the description of the artificial neural network into instructions for implementation on a deep learning accelerator. For example, the method can be implemented on a computing device to generate DLA instructions (205) and a DLA matrix (207) for use in... Figure 1 The described integrated circuit device (101) or Figure 5 The system described implements matrix computation using an artificial neural network (201).

[0132] After the computing device receives the description (221) of the artificial neural network (201), the computing device generates a first result of compilation from the description (221) of the artificial neural network (201) according to the specifications of the first device.

[0133] For example, the specification of the first device could be the general DLA specification (225); and the first result of the compilation could be Figures 6 to 9 The general result described (227) is the result of the DLA compiler (203) performing compilation and optimization (223) according to the general DLA specification (225).

[0134] The first result may include first data representing first instructions that can be executed on the first device to perform matrix computations of an artificial neural network (201) according to the specifications of the first device.

[0135] For example, the first instruction executable on the first device may include general DLA instructions (e.g., 241) and / or general routines (e.g., 245) used in the general result (227) to implement the computation of the artificial neural network (201) on the general deep learning accelerator. The general deep learning accelerator may be a virtual device according to the general DLA specification (225), or a reference implementation of the general DLA specification (225).

[0136] The computing device maps the first result of compilation to the second result according to the specifications of the second device.

[0137] For example, the specification of the second device can be a specific DLA specification (235); and the second result can be Figure 7 The compiler output described in (237), or Figure 8 The mapped result described in (229). For example, the second device can be... Figure 8 An integrated circuit device (101) having Figures 2 to 4 The matrix processing unit described in the document.

[0138] The second result may include second data representing second instructions that can be executed on the second device to implement matrix computation of an artificial neural network (201).

[0139] For example, the second instruction may be a DLA instruction (205) according to a specific DLA specification (235). The second instruction may contain a DLA routine (e.g., 243 and / or 247).

[0140] The computing device can further generate third data representing the parameters of the artificial neural network (201) from the description (221) of the artificial neural network (201).

[0141] For example, third data representing the parameters of the artificial neural network (201) may include a DLA matrix (207). Some DLA matrices (207) may be loaded into kernel buffers (131 to 133) in the processing unit (111) of the integrated circuit device (101). Some DLA matrices (207) may be loaded into mapped memory (151 to 153) in the processing unit (111) of the integrated circuit device (101).

[0142] For example, the second device can be Figure 1 An integrated circuit device (101) having a random access memory (105) configured to store third data representing parameters of an artificial neural network and second data representing a second instruction. Figure 1 The integrated circuit device (101) further includes at least one processing unit (111) configured to execute a second instruction to generate an output (213) of the artificial neural network (201) based on a third data representing parameters of the artificial neural network (201) and a fourth data representing input (211) to the artificial neural network (201).

[0143] like Figure 7 and 8 As explained, mapping the first result to the second result may include mapping instructions in the first result that can be executed by the first device to routines in the second result that can be executed by the second device. For example, a general DLA instruction (241) in the general result (227) may be mapped to a DLA routine (243) executable by a deep learning accelerator (103) of a specific platform, as identified by a specific DLA specification (235). Preferably, the DLA routine (243) may be pre-optimized to perform the task defined by the general DLA instruction (241).

[0144] like Figure 8 As explained, mapping the first result to the second result may include mapping a combination of instructions executable by the first device in the first result to a routine executable by the second device in the second result. For example, the combination of instructions may be a general routine (245) mapped to a corresponding DLA routine (247) during the operation of DLA mapping (233). Preferably, the corresponding DLA routine (247) may be pre-optimized to perform the task defined by the combination of instructions (e.g., the general routine (245)).

[0145] Optional, such as Figure 9 As explained, the DLA compiler (203) can further transform the second result into a third result having fifth data representing a third instruction that can be executed in the second device.

[0146] For example, the second result may include Figure 9 The mapping result described is (229); and the third result can be... Figure 9 The compiler output (237) is described. The DLA compiler (203) performs optimizations (231) in the transformation, such that when executed in a deep learning accelerator (103) that conforms to or is based on a specific DLA specification (235), the DLA instructions (205) compiled in the compiler output (237) have better performance than the instructions compiled in the mapped result (229).

[0147] Optionally, the computing device may store third data representing parameters of the artificial neural network (201) and second data representing a second instruction (or fifth data representing a third instruction) in the random access memory (105) of the integrated circuit device (101). Furthermore, the computing device or another device may store fourth data representing input (211) to the artificial neural network (201) in the random access memory (105) of the integrated circuit device (101) to cause the integrated circuit device (101) to execute the second instruction (or the third instruction) and generate the output (213) of the artificial neural network (201).

[0148] Figure 10 An integrated circuit device having a deep learning accelerator with configurable hardware capabilities and random access memory is shown according to one embodiment.

[0149] exist Figure 10 In this context, the processing unit (111) can be used in different configurations. Different configurations of the processing unit (111) offer different trade-offs in terms of functionality, efficiency, performance, and / or energy consumption.

[0150] A set of registers (251) is provided to control the circuit configuration currently available for performing computations. A set of hardware options (253) specified in registers (251) selects the circuit configuration used by the processing unit (111) when executing instructions for matrix operations.

[0151] When a set of hardware options (253) is stored in register (251), the processing unit (111) is configured to operate during data processing according to one of a plurality of designs of circuit configuration. When another set of hardware options (253) is stored in register (251), the processing unit (111) is configured to operate according to another of a plurality of designs. Thus, at least some computational aspects of the processing unit (111) can be configured or selectively used by specifying the hardware options (253) in the register.

[0152] For example, in one embodiment, the deep learning accelerator (103) may have hardware options configured to control the granularity of matrix computation.

[0153] For example, based on the hardware options specified in register (251), a vector-vector unit (e.g., 161) may be configured to compute the sum of products of elements from its vector buffer, or the sum of products of the first half of the elements from its vector buffer, or the sum of products of the second half of the elements from its vector buffer, or a combination thereof.

[0154] For example, based on the hardware options specified in register (251), the matrix-vector unit (e.g., 141) can be configured to compute the product of a matrix (e.g., stored in a set of mapped stores (151, ..., 153)) and a vector (e.g., stored in a kernel buffer (131)), or the product of a portion of a matrix and a portion of a vector, or the product of another portion of a matrix and another portion of a vector, or a combination thereof.

[0155] For example, based on the hardware options specified in register (251), the matrix-vector unit (e.g., 141) can be configured to compute the product of a matrix (e.g., stored in a set of mapped stores (151, ..., 153)) and a vector (e.g., stored in a kernel buffer (131)), or the product of a portion of a matrix and a portion of a vector, or the product of another portion of a matrix and another portion of a vector, or a combination thereof.

[0156] For example, based on the hardware options specified in register (251), a matrix-matrix unit (e.g., 121) can be configured to compute the product of a matrix (e.g., stored in a set of mapped stores (151, ..., 153)) and another matrix (e.g., stored in a set of kernel buffers (131, ..., 133)), or the product of parts of the matrices, or the product of alternative parts of the matrices, or a combination thereof.

[0157] Therefore, the hardware option (253) can be used to adjust the granularity level of the processing unit (111) and organize the concurrent execution of parallel units (e.g., matrix-vector units (141, ..., 143) in matrix-vector units (121), vector-vector units (161, ..., 163) in matrix-vector units (141) and / or multiplication-accumulation units (171, ..., 173) in vector-vector units (161)).

[0158] When different option groups are specified in register (251), the processing unit (111) is effectively configured to have different hardware capabilities for matrix computation.

[0159] Figure 11 This describes different hardware configurations of the processing unit of a deep learning accelerator, configurable via options stored in registers, according to one embodiment. For example, Figure 11 The processing unit can be used Figure 1 , 5In and / or in a deep learning accelerator of 10 (103).

[0160] exist Figure 11 In this process, the processing unit (255) is controlled by the register (257).

[0161] In configuration A (265), option A (261) is specified in register (257). Option A (261) in register (257) enables processing unit (255) to function as processing unit A (259) in terms of functionality and / or performance.

[0162] When option A (261) in register (257) is changed to option B (263), the combination of processing unit (255) and register (257) is in configuration B (267). Option B (263) in register (257) enables processing unit (255) to function as processing unit B (269) in terms of functionality and / or performance.

[0163] Processing unit A (259) and processing unit B (269) differ in functionality and / or performance. In some computing tasks or scenarios, using processing unit A (259) may be more advantageous than using processing unit B (269), but not in others. Options (261 and 263) are optionally stored in register (257) to selectively configure or convert processing unit (255) into processing unit A (259) and processing unit B (269). Therefore, processing units (259 and 269) can be selectively deployed in different configurations (265 and 267) for different computing tasks or scenarios.

[0164] Figure 12 This describes a technique, according to one embodiment, for generating instructions executable by a deep learning accelerator with optimized hardware configuration to implement an artificial neural network.

[0165] The DLA compiler (203) initially transforms the ANN description (221) into a general result by compiling and optimizing (223) according to the general DLA specification.

[0166] For example, an ANN description (221) can identify various aspects of an artificial neural network (201), including behavioral models of artificial neurons and connectivity of artificial neurons in the network. Parameters used in the ANN description (221) may include the identification of activation functions, biases, and / or states of artificial neurons. Furthermore, parameters may include synaptic weights of connections between artificial neurons. The description (221) may be specified in a standard format (e.g., Open Neural Network Exchange (ONNX)) and provided as input to a DLA compiler (203).

[0167] The general DLA specification (225) identifies the computational capabilities of general-purpose deep learning accelerators. Therefore, compilation and optimization (223) are independent of the hardware platform or capabilities of the deep learning accelerator.

[0168] The general result (227) may contain instructions for matrix computation of artificial neural networks (201) on a general or virtual deep learning accelerator that conforms to the general DLA specification (225).

[0169] Subsequently, the DLA compiler (203) can map the general result (227) to the mapped result (229) through the operation of the DLA mapping (233). The DLA mapping (233) is based on a specific DLA specification (235) that identifies the hardware capabilities of a specific hardware platform for the deep learning accelerator.

[0170] exist Figure 12 In this context, deep learning accelerators based on specific DLA specifications (235) have configurable hardware options (253), such as... Figure 10 and 11 As explained, the DLA compiler (203) uses a set of default options to convert the general result (227) into the mapped result (229) when performing the DLA mapping (233). Therefore, the instructions in the mapped result (229) are configured to use the deep learning accelerator (103), wherein the set of default hardware options (253) is stored in its register (251).

[0171] For example, it can be used Figure 7 and 8 The technology is used to perform DLA mapping (233). The DLA compiler (203) can map a general result (227) compiled for a general deep learning accelerator to a mapped result (229) that can be executed on a specific platform of the deep learning accelerator, which is configured using the set of default hardware options (253) in its register (251).

[0172] Following the DLA mapping (233), the DLA compiler (203) may further perform optimization (231) on the compiled mapped result (229) to produce compiler output (237). During optimization (231), the DLA compiler (203) may selectively tune hardware options (253) to improve the performance of the deep learning accelerator when implementing the artificial neural network (201) specified by the ANN description (221).

[0173] For example, the DLA compiler (203) can perform optimizations (231) by reducing the energy consumption and / or computation time used when executing DLA instructions (205) in the deep learning accelerator (103). A set of optimized hardware options (253) can be used to specifically optimize the hardware of the deep learning accelerator (103) for implementing a particular artificial neural network (201) specified by the ANN description (221). After the integrated circuit device (101) is manufactured, hardware optimizations can be implemented by storing the set of optimized hardware options (253) in a register (251).

[0174] For example, the DLA instruction (205) may include instructions for storing a set of optimized hardware options (253) into a register (251) during an initialization operation to configure the deep learning accelerator (103) to execute the remainder of the DLA instruction (205).

[0175] In some implementations, hardware options (253) can be adjusted during the execution of the DLA instruction (205). For example, a first set of hardware options (253) in register (251) can be used to execute the first part of the DLA instruction (205); and a second set of hardware options (253) in register (251) can be used to execute the second part of the DLA instruction (205).

[0176] In some implementations, the DLA instructions (205) do not contain instructions that modify the contents of registers (251). During the operation of loading compiler output (237) into the random access memory (105) of the integrated circuit device (101) to configure the computation of the artificial neural network (201), the host system of the integrated circuit device (101) loads the hardware options (253) selected by the DLA compiler (203) into registers (251). Therefore, the deep learning accelerator (103) is configured to execute DLA instructions (205) optimized for the hardware options (253) to perform the computation of the artificial neural network (201).

[0177] In one embodiment of the method, a DLA compiler (203) is configured to generate and optimize instructions for a deep learning accelerator with configurable hardware options. The method can be used with... Figure 1 , 5 The deep learning accelerator (103) described in 10 to 11 generates instructions and selects hardware options for implementing computations of the artificial neural network (201).

[0178] In the method, the computing device receives a description (221) of an artificial neural network (201). Furthermore, the computing device generates a first compiled result from the description (221) of the artificial neural network (201) according to the specifications (235) of the first device.

[0179] For example, the first result could be Figure 12 The mapping result (229) is obtained by compilation and optimization (223) according to the general DLA specification (225) and DLA mapping according to the specific DLA specification (235).

[0180] For example, the first device may be at least one processing unit (e.g., 111, 141 or 255) configured to perform matrix calculations and having a hardware configuration (e.g., 265 and 267) selectable via at least one register (e.g., 257).

[0181] For example, the function of a processing unit (e.g., 111, 141, or 255) can be adjusted based on the contents stored in at least one register (e.g., 257). Figure 11 As described herein, when a first set of hardware options (e.g., 261) is specified in at least one register (e.g., 257), the processing unit (e.g., 111, 141, 255) is configured to perform a first function of processing unit A (259); and when a second set of hardware options (e.g., 263) is specified in at least one register (e.g., 257), the processing unit (e.g., 111, 141, 255) is configured to perform a second function of another processing unit B (259) that is different from the first function.

[0182] Furthermore, the computing device transforms the first result of the compilation into a second result to select the hardware options of the first device (e.g., 253).

[0183] For example, the second result could be Figure 12 The compiler output (237) described herein. The second result may contain first data representing parameters of the artificial neural network, such as a DLA matrix (207). The second result may further contain second data representing instructions executable by at least one processing unit of the first device to produce the output (213) of the artificial neural network (201) in response to third data representing input (211) to the artificial neural network (201). The second result may further contain fourth data representing hardware options (e.g., 253) to be stored in at least one register (e.g., 257) to configure at least one processing unit (e.g., 111, 141, or 255).

[0184] In one implementation, the contents of at least one register (e.g., 251, 257) can be updated by executing part of an instruction representing a second data stored in a random access memory (105) connected to the deep learning accelerator (103).

[0185] For example, at least one interface (e.g., 107) of the integrated circuit device (101) may be configured to receive third data as input (211) to the artificial neural network (201) and store the third data in a random access memory (105).

[0186] Before executing an instruction (205) represented by second data stored in random access memory (105), the contents stored in at least one register (251) can be updated via at least one interface (e.g., 107). Therefore, the contents of at least one register (251) will not change during the execution of the DLA instruction (205) generated by the DLA compiler (203).

[0187] Alternatively, the contents stored in at least one register (251) can be dynamically updated via a portion of the instructions of the deep learning accelerator (103). For example, the processing unit (255) can operate on some DLA matrices (207) using configuration A (265) and on other DLA matrices (207) using configuration B (267).

[0188] For example, the dimensions of the two matrix operands of the instruction to be processed by the processing unit (255) can be configured according to at least one register (257) used to execute the instruction in the processing unit (255).

[0189] Optionally, the DLA compiler (203) may optimize the instructions generated for implementing the artificial neural network (201) in the deep learning accelerator (103) based not only on hardware configuration options but also on the overall hardware characteristics of the deep learning accelerator (103). Such hardware characteristics of the deep learning accelerator (103) may include the characteristics, options, behavior, performance, latency, and / or limitations of the deep learning accelerator (DLA). The DLA compiler (203) can learn the hardware characteristics of the deep learning accelerator (DLA) by interacting with it. For example, the DLA compiler (203) may test the hardware of the deep learning accelerator (103) based on the computational performance of executing different instructions, different combinations of instructions, different instruction execution scenarios, and combining different hardware configuration options. For example, the DLA compiler (203) may optionally train the hardware of the deep learning accelerator (DLA) by running test instruction sets to discover and / or measure hardware capabilities, options, performance, latency, etc. Optionally, the DLA compiler (203) can use artificial neural networks to determine the characteristics, options, behavior, performance, latency, and / or limitations of the deep learning accelerator (DLA) through machine learning. Information about the hardware characteristics of the deep learning accelerator (DLA) can be used in the compiler's optimization operations to generate or customize the instructions and / or runtime for implementing the artificial neural network (201) on the deep learning accelerator (DLA).

[0190] Figure 13 This demonstrates a technique for discovering hardware features of deep learning accelerators according to one embodiment.

[0191] exist Figure 13 In this embodiment, a computing device (271) equipped with a DLA compiler (203) is connected to an integrated circuit device (101) to learn its hardware characteristics using a test program (273). The execution result of the test program (273) can be retrieved from random access memory (105) as a response (275) to the test program. The response (275) can be analyzed to identify the hardware characteristics (279) of the deep learning accelerator (103).

[0192] For example, the DLA compiler (203) can generate one or more test programs (273) according to the DLA specification (235) of the deep learning accelerator (103). Some test programs can be generated based on instances of artificial neural networks (e.g., 201). Optionally, or in combination, some test programs can be generated to train various processing units (111) of the deep learning accelerator (103) without modeling a specific artificial neural network.

[0193] For example, the test program (273) can be written into the random access memory (105) in a manner similar to how the DLA instructions (205) and DLA matrix (207) of the artificial neural network (201) are written into the random access memory (105) for execution by the deep learning accelerator (103). Furthermore, the execution of the test program (273) can be triggered by writing input to the random access memory (105), similar to providing input (211) to the artificial neural network (201) in the random access memory (105) to trigger the execution of the DLA instructions (205) of the artificial neural network (201).

[0194] The response (275) may include timestamps of the execution of various stages of the test program (273). The timestamps can be used to measure the execution time of instructions between timestamps, and thus measure the performance characteristics of the processing unit (111) of the deep learning accelerator (103) in executing specific instructions and / or specific instruction sequences.

[0195] In some implementations, the integrated circuit device (101) is capable of generating timestamps during the execution of instructions stored in random access memory (105); and the test program can be configured by the DLA compiler (203) to include instructions that guide the integrated circuit device (101) to generate timestamps.

[0196] In some implementations, the computing device (271) generates a timestamp for triggering the execution of a test program (273) stored in random access memory (105) and a timestamp for receiving a response (275) generated by the execution of the test program (273). The timestamps measure the performance of the deep learning accelerator (103) while executing the test program (273). Performance measurements of different test programs (273) can be used to determine the performance differences in instructions within the test programs (273).

[0197] For example, two test programs may share a common instruction set, while one of the test programs has an additional instruction set. Therefore, the difference in execution time between the two test programs provides a performance measurement of the deep learning accelerator (103) when executing the additional instruction set. The additional instruction set can be repeated multiple times in the corresponding test program to improve the accuracy of the performance measurement. Therefore, multiple test programs can be constructed to measure the performance level of various instructions that can be executed in the processing unit (111).

[0198] For example, when the deep learning accelerator (103) has different hardware options (253), the DLA compiler (203) can use different hardware options (253) to generate different routines to perform the same task. The performance level of the routines can be measured to identify the optimal routines for the task and the corresponding hardware options (253). The optimal routines can be configured at runtime or in a library and used to implement artificial neural networks (201) in the deep learning accelerator (103).

[0199] Optionally, the DLA compiler (203) running in the computing device (271) is configured to identify the specifications and / or DLA hardware characteristics (279) of the deep learning accelerator (103) connected to the computing device (271). For example, based on a set of specifications for a known hardware platform of the deep learning accelerator, the DLA compiler (203) may build a test program (273) for execution by the deep learning accelerator (103) connected to the computing device (271). Based on the computation results in the response (275), the DLA compiler (203) may determine which specification the deep learning accelerator (103) connected to the computing device (271) conforms to. Optionally, after identifying the specifications of the deep learning accelerator (103), the DLA compiler (203) may build a test program (273) to verify and confirm that the functionality of the deep learning accelerator (103) currently conforms to the specifications.

[0200] In some cases, the hardware platform of the deep learning accelerator (103) connected to the computing device (271) may be new to the DLA compiler (203). The deep learning accelerator (103) is not manufactured according to any specifications known to the DLA compiler (203). In this case, the DLA compiler (203) can use a test program to probe the functionality of the deep learning accelerator (103). The probing operation can be used to discover functionality similar to some known platforms. The result of the probing can produce specifications for at least a subset of the functionality of the deep learning accelerator (103). When the subset is sufficient to implement the computation of the artificial neural network (201), the DLA compiler (203) can produce compiler output (237) for implementing the artificial neural network (201), even if the full and / or official specifications of the deep learning accelerator (103) are not available.

[0201] Figure 14 This describes a technique, according to one embodiment, for generating instructions that can be executed by a deep learning accelerator and optimized according to the hardware characteristics of the deep learning accelerator.

[0202] exist Figure 14 In this process, DLA training (277) is constructed and / or performed to determine DLA hardware characteristics (279) to support optimization (231) in compiling the ANN description (221) of the hardware platform of the deep learning accelerator identified by the specific DLA specification (235).

[0203] For example, DLA training (277) may include running a test program (273) in a deep learning accelerator (103) according to a specification (235) and analyzing the response (275) of the deep learning accelerator (103) to the test program (273), such as Figure 13 As explained.

[0204] Optionally, the DLA hardware features (279) obtained from the DLA training (277) may be included in the specifications (235) of the deep learning accelerator (103).

[0205] exist Figure 14 In the process, the DLA compiler (203) initially transforms the ANN description (221) into a general result (227) by compilation and optimization (223) according to the general DLA specification (225).

[0206] The ANN description (221) may include a behavioral model of the artificial neurons and the connectivity of the artificial neurons in the network. The ANN description (221) may include the activation function, bias, and / or state identification of the artificial neurons. The ANN description (221) may include synaptic weights used for connections between artificial neurons.

[0207] The DLA compiler (203) transforms a description (221) of a standard format (e.g., Open Neural Network Exchange (ONNX)) into a general result (227) that can be implemented on a deep learning accelerator that conforms to the general DLA specification (225).

[0208] For example, the general result (227) may include DLA instructions and a DLA matrix for implementing computation of the artificial neural network (201) on a virtual deep learning accelerator configured according to the general DLA specification (225). For example, the virtual deep learning accelerator may be implemented via software running on a microprocessor and / or computing device (271).

[0209] Subsequently, the DLA compiler (203) can map the general result (227) to the mapped result (229) through the operation of the DLA mapping (233). The DLA mapping (233) is based on a specific DLA specification (235) that identifies the hardware capabilities of a specific hardware platform of the deep learning accelerator (e.g., a deep learning accelerator (103) connected to a computing device (271) in which the DLA compiler (203) runs).

[0210] exist Figure 14 In this context, deep learning accelerators according to specific DLA specifications (235) may optionally have configurable hardware options (253), such as... Figure 10 and 11 As explained, the DLA compiler (203) uses a set of default options to convert the general result (227) into the mapped result (229) when performing the DLA mapping (233). Therefore, the instructions in the mapped result (229) are configured to use the deep learning accelerator (103), wherein the set of default hardware options (253) is stored in its register (251).

[0211] For example, it can be used Figure 7 and 8 The technology is used to perform DLA mapping (233). The DLA compiler (203) can map a general result (227) compiled for a general deep learning accelerator to a mapped result (229) that can be executed on a specific platform of the deep learning accelerator, which is configured using the set of default hardware options (253) in its register (251).

[0212] Following the DLA mapping (233), the DLA compiler (203) may further perform optimizations (231) on the compiled mapped result (229) to produce compiler output (237). During optimization (231), the DLA compiler (203) may selectively adjust hardware options (253) and / or instruction sequences or routines based on DLA hardware characteristics (279) to improve the performance of the deep learning accelerator in implementing the artificial neural network (201) specified by the ANN description (221).

[0213] For example, computational tasks can be implemented using different sequences of instructions (optionally using different hardware options to run). Based on hardware characteristics (279), the DLA compiler (203) can calculate or estimate the time required to execute different sequences and select the sequence with the best performance among the different sequences. The DLA compiler (203) can optionally incrementally transform the instructions to reduce the computation / estimation execution time during optimization (231).

[0214] Figure 15 This paper demonstrates a method for implementing artificial neural networks on a deep learning accelerator based on the hardware characteristics of the deep learning accelerator, according to one embodiment, using compilation instructions.

[0215] For example, Figure 15 The method can be run Figure 14 The DLA compiler (203) described herein is implemented in the computing device (271).

[0216] In block 301, the computing device (271) interacts with the integrated circuit device (101) to identify the hardware characteristics (279) of the integrated circuit device (101) when performing matrix calculations.

[0217] For example, the integrated circuit device (101) may have Figure 1 Or the deep learning accelerator (103) described in section 10. The deep learning accelerator (103) may have, for example, the deep learning accelerator (103) described in section 10. Figures 1 to 4 One or more processing units (e.g., 111, 121, 141, 161) are described in the document to perform matrix calculations.

[0218] For example, the hardware characteristics (279) of the integrated circuit device identify the features, options, behaviors, performance, or delays of at least one processing unit (e.g., 111, 121, 141, 161) of the integrated circuit device (101), or any combination thereof.

[0219] For example, such as Figure 13As explained, the computing device (271) can interact with the integrated circuit device (101) by generating a test program and loading it into the integrated circuit device (101) and receiving a response (275) from executing the test program in the integrated circuit device (101).

[0220] For example, the computing device (271) may determine hardware characteristics (279) from the response (275), including the calculation results of the test program (273) and / or the timestamp associated with the execution of the test program (273) in the integrated circuit device (101).

[0221] For example, the computing device (271) can identify the specification of the integrated circuit device (101) among a plurality of predetermined specifications of the integrated circuit device based on the response (275) of executing a test program in the integrated circuit device. For example, identification can be made when the result of the test program (273) matches the result produced by an integrated circuit device that conforms to the identified specification, but does not match the result produced by an integrated circuit device that conforms to other specifications.

[0222] Optionally, the computing device (271) can probe the hardware options (253) of the processing units (e.g., 111, 121, 141, 161) of the integrated circuit (101) by trial and error and based on the response (275) of executing a test program (273) in the integrated circuit device (101).

[0223] For example, the computing device (271) may generate test programs based on the specifications of multiple hardware platforms of an integrated circuit device configured to perform matrix calculations. In determining hardware characteristics (279), the computing device (271) may examine responses (275) to determine which instructions take effect with what delay and / or perform what functions, etc.

[0224] In box 303, computing device (271) receives description (221) of artificial neural network (201).

[0225] In box 305, the DLA compiler (203) running in the computing device (271) produces a compilation result from the description (221) of the artificial neural network (201) based at least in part on the hardware characteristics (279) of the integrated circuit device (101).

[0226] For example, the result of compilation may include first data representing the parameters of the artificial neural network (201), such as a DLA matrix (207). The result of compilation may further include second data representing instructions (205) that can be executed by the integrated circuit device (101) to generate the output (213) of the artificial neural network (201) based on the first data representing the parameters of the artificial neural network (201) and based on the input (211) to the artificial neural network (201).

[0227] For example, the DLA compiler (203) can transform the compilation result from the description (221) of the artificial neural network (201) to improve the performance of the compilation result when it is run or executed in the integrated circuit device (101) based on the hardware characteristics (279) of the integrated circuit device (101). For example, such as Figure 14 As explained in the document, optimizations (231) can be performed based on hardware options (253) and / or hardware features (279).

[0228] For example, the computing device (271) can use Figure 16 The machine implementation described in the document.

[0229] Figure 16 This describes an example machine of a computer system, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0230] In some embodiments, Figure 16 Computer systems can be used with Figures 2 to 4 The matrix processing unit described in the document Figure 1 The integrated circuit device (101) is implemented Figure 5 The system.

[0231] Figure 16 The computer system can be used to execute reference instructions configured to perform operations corresponding to the DLA compiler (203). Figures 1 to 15 The operation of the DLA compiler (203) is described.

[0232] In some embodiments, the machine may be connected (e.g., networked) to other machines on a local area network (LAN), intranet, extranet, and / or the Internet. The machine may operate as a server or client machine in a client-server network environment, as a peer-to-peer (or distributed) network environment, or as a server or client machine in a cloud computing infrastructure or environment.

[0233] For example, a machine can be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, network device, server, network router, switch, or bridge, or any machine capable of (sequentially or otherwise) executing a set of instructions specifying actions to be taken by said machine. Furthermore, while a single machine is described, the term "machine" should also be considered to include any collection of machines that individually or jointly execute a set (or more) of instructions to perform any or more of the methods discussed herein.

[0234] Figure 16The example computer system described herein includes a processing unit (402), a main memory (404), and a data storage system (418) that communicate with each other via a bus (430). For example, the processing unit (402) may include one or more microprocessors; the main memory may include read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), static random access memory (SRAM), etc. The bus (430) may include multiple buses or be replaced by multiple buses.

[0235] Figure 16 The processing device (402) in the text represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or multiple processors implementing combinations of instruction sets. The processing device (402) may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, or the like. The processing device (402) is configured to execute instructions (426) for performing the operations discussed in conjunction with the DLA compiler (203). Optionally, the processing device (402) may include a deep learning accelerator (103).

[0236] Figure 16 The computer system may further include a network interface device (408) for communication via a computer network (420).

[0237] Optionally, the bus (430) is connected to an integrated circuit device (101) having Figure 1 And / or the deep learning accelerator (103) and random access memory (105) described in 10. The compiler (203) may write its compiler output (237) into the random access memory (105) of the integrated circuit device (101) to enable the integrated circuit device (101) to perform matrix calculations of the artificial neural network (201) specified by the ANN description (221). Optionally, the compiler output (237) may be stored in the random access memory (105) of one or more other integrated circuit devices (101) via a network interface device (408) and a computer network (420).

[0238] The data storage system (418) may include a machine-readable medium (424) (also referred to as a computer-readable medium) thereon storing one or more sets of instructions (426) or software embodying any or more of the methods or functions described herein. The instructions (426) may also reside wholly or at least partially in main memory (404) and / or processing device (402) during execution by a computer system, which also constitute machine-readable storage media.

[0239] In one embodiment, the instructions (426) include instructions for implementing the DLA compiler (203) (e.g., refer to...). Figures 5 to 15 The DLA compiler (203) described herein contains functional instructions. Although the machine-readable medium (424) is shown as a single medium in the exemplary embodiments, the term "machine-readable storage medium" should be considered to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for machine execution and causing the machine to perform any or more of the methods of this disclosure. Therefore, the term "machine-readable storage medium" should be considered to include (but is not limited to) solid-state memory, optical media, and magnetic media.

[0240] This disclosure includes methods and apparatus for performing the methods, the apparatus including a data processing system for performing the methods, and a computer-readable medium containing instructions that, when executed on the data processing system, cause the system to perform the methods.

[0241] A typical data processing system may include interconnects (e.g., buses and system core logic) that interconnect (multiple) microprocessors and memory. Microprocessors are typically coupled to cache memory.

[0242] Interconnectors link together microprocessors and memory and also interconnect them to I / O devices via input / output (I / O) controllers. I / O devices may include display devices and / or peripheral devices such as mice, keyboards, modems, network interfaces, printers, scanners, cameras, and other devices known in the art. In one embodiment, when the data processing system is a server system, some I / O devices (e.g., printers, scanners, mice, and / or keyboards) are optional.

[0243] Interconnectors may include one or more buses interconnected with each other via various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling USB (Universal Serial Bus) peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0244] The memory can include one or more of the following: ROM (Read Only Memory), volatile RAM (Random Access Memory), and non-volatile memory, such as hard disk drive, flash memory, or the like.

[0245] Volatile RAM is typically implemented as dynamic RAM (DRAM) which requires power continually in order to refresh or maintain the data in the memory. Non-volatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., a DVD Redwriter), or other type of memory system which maintains data even after power is removed from the system. The non-volatile memory can also be a random access memory.

[0246] The non-volatile memory can be a local device coupled directly to a remaining components in the data processing system. A non-volatile memory in a system can also be used, such as a network storage device which is coupled to the data processing system through a network interface, such as an Ethernet or modem interface.

[0247] In this disclosure, some functions and operations are described as being performed by or caused by software code to simplify description. However, such expressions are also used to specify that the functions result from execution of the code / instructions by a processor, such as a microprocessor.

[0248] Alternatively, or in combination, the functions and operations described herein can be implemented using special purpose circuitry, either in hardware alone or in combination with software instructions, such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Embodiments can be implemented using hardwired circuitry, without software instructions, or in combination with software instructions. Accordingly, the technology is not limited to any particular combination of hardware circuitry and software.

[0249] While one embodiment can be implemented in fully functioning computers and computer systems, various embodiments can be distributed over computer systems at various sites, and the embodiments can be implemented in conjunction with a variety of computer systems and computer architectures.

[0250] At least some aspects of the disclosure can be embodied and / or implemented at least in part in software. In other words, the technology can be embodied at least in part in software programs, firmware, coding, instructions, instruction sets, code segments, program configurations, routines, subroutines, programs, applications, software modules, microcode, microinstructions, microprograms, software drivers, firmware drivers, firmware modules, software modules, instructions, operation-specific circuitry, application-specific circuitry, and / or video circuitry, etc., which execute from various computer media.

[0251] The routines executed to implement the embodiments may be implemented as part of an operating system or as part of a particular application, component, program, object, module, or a sequence of instructions referred to as a "computer program." A computer program typically contains one or more instructions set at different times in various memories and storage devices of a computer, and when read and executed by one or more processors in the computer, the instructions cause the computer to perform operations necessary to carry out elements relating to various aspects.

[0252] Machine-readable media can be used to store software and data that enable a data processing system to perform various methods when executed. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache. Parts of this software and / or data can be stored in any of these storage devices. Furthermore, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions, or in the same communication session. Data and instructions can be obtained in their entirety before the application is executed. Alternatively, portions of data and instructions can be obtained dynamically and in a timely manner as needed for execution. Therefore, it is not required that all data and instructions be on the machine-readable media at any given moment.

[0253] Examples of computer-readable media include (but are not limited to) media of non-transitory, recordable, and non-recordable types, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, disk storage media, optical storage media (e.g., optical disc read-only memory (CD-ROM), digital versatile disk (DVD), etc.) and others. Computer-readable media can store instructions.

[0254] Instructions can also be embodied in digital and analog communication links used for electrical, optical, acoustic, or other forms of propagated signals (such as carrier waves, infrared signals, digital signals, etc.). However, propagated signals such as carrier waves, infrared signals, digital signals, etc., are not tangible machine-readable media and are not configured to store instructions.

[0255] Generally, machine-readable media includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible to a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device having one or more processors, etc.).

[0256] In various embodiments, hardwired circuitry systems can be used in combination with software instructions to implement the technology. Therefore, the technology is neither limited to any particular combination of hardware circuitry systems and software, nor to any particular source of instructions for execution by a data processing system.

[0257] The above description and accompanying drawings are illustrative and not intended to be limiting. Many specific details have been described to provide a thorough understanding. However, in certain instances, well-known or conventional details have not been described to avoid obscuring the description. References to one or more embodiments of this disclosure are not necessarily references to the same embodiment; and such references imply at least one.

[0258] In the foregoing description, this disclosure has been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope set forth in the appended claims. Therefore, this specification and the accompanying drawings are to be regarded in an illustrative sense rather than a restrictive sense.

Claims

1. A method comprising: A computing device transmits one or more commands to an integrated circuit device, each command being associated with one or more requests in response to a hardware feature of the integrated circuit device when performing matrix calculations; At the computing device and after determining the hardware characteristics of the integrated circuit device, data representing a description of the artificial neural network is received; and The computing device generates a compilation result from the data representing the description of the artificial neural network, based at least in part on the hardware characteristics of the integrated circuit device. The compilation result includes first data and second data, the first data representing parameters of the artificial neural network, and the second data representing instructions that can be executed by the integrated circuit device to generate the output of the artificial neural network based on the first data and the input to the artificial neural network.

2. The method of claim 1, wherein the hardware characteristics of the integrated circuit device identify features, options, behaviors, performance, or delays of at least one processing unit of the integrated circuit device, or any combination thereof.

3. The method of claim 2, wherein transmitting the one or more commands to the integrated circuit device comprises loading a test program into the integrated circuit device to receive a response that the test program is executed in the integrated circuit device.

4. The method of claim 3, further comprising: The hardware characteristics are determined from the response of the test program executed in the integrated circuit device.

5. The method of claim 3, further comprising: The hardware characteristics are determined from the timestamps associated with the execution of the test program in the integrated circuit device.

6. The method of claim 3, further comprising: Based on the response of the test procedure executed in the integrated circuit device, the specifications of the integrated circuit device are identified among a plurality of predetermined specifications of the integrated circuit device.

7. The method of claim 3, further comprising: The hardware options of the matrix processing unit of the integrated circuit are probed based on the response of the test procedure executed in the integrated circuit device.

8. The method of claim 3, further comprising: The test program is generated based on the specifications of multiple hardware platforms configured to perform matrix calculations by integrated circuit devices.

9. The method of claim 3, further comprising: The compilation result derived from the description of the artificial neural network is transformed to improve the performance of the result when executed in the integrated circuit device, based on the hardware characteristics of the integrated circuit device.

10. A computing device comprising: Memory; and At least one microprocessor coupled to the memory and configured to probe an integrated circuit device to identify hardware characteristics of the integrated circuit device when performing matrix calculations, and to generate a compilation result from data representing a description of an artificial neural network, based at least in part on the hardware characteristics of the integrated circuit device, the compilation result comprising first data and second data, the first data representing parameters of the artificial neural network, and the second data representing instructions executable by the integrated circuit device to generate the output of the artificial neural network based on the first data and inputs to the artificial neural network.

Images