Apparatus for and mechanism for processing neural network task using single chip package having a plurality of identical dies

Abstract

To provide an artificial intelligence processing unit for and a method of processing a neural network model.SOLUTION: An artificial intelligence processing unit has a plurality of identical artificial intelligence processing dies 103a to 103f. For example, the artificial intelligence processing die 103a has at least one inter-die input block 109a, 109b, and at least one inter-die output block 111a, 111b. In each of the artificial intelligence processing dies, dies from the at least one inter-die output block (e.g., the inter-die output block 111a) of the artificial intelligence processing die (e.g., artificial intelligence processing die 103a) to at least one inter-die input block (e.g., inter-die input block 221a) of the artificial intelligence processing die (e.g., the artificial intelligence processing die 103b) are connected to one another through one or more communication paths so as to be capable of communicating with one another. Each of the artificial intelligence processing dies corresponds to at least one layer of a neural network.SELECTED DRAWING: Figure 2A

Description

[Background technology]

[0001] Related Applications This application claims priority to and the benefit of U.S. patent application Ser. No. 15 / 819,753, filed Nov. 21, 2017, and entitled "APPARATUS AND MECHANISM FOR PROCESSING NEURAL NETWORK TASKS USING A SINGLE CHIP PACKAGE WITH MULTIPLE IDENTICAL DIES," the entire contents of which are incorporated herein by reference for all purposes.

[0002] background The use of neural networks in the field of artificial intelligence computing has grown rapidly over the past few years. More recently, the use of special-purpose computers, such as application-specific integrated circuits (ASICs), has been used to process neural networks. However, the use of ASICs poses several challenges. Some of these challenges are (1) long design times and (2) significant non-recurring engineering costs. As neural networks grow in popularity and the range of tasks for which they are used expands, the long design times and significant non-recurring engineering costs become worse. Summary of the Invention

[0003] overview At least one aspect is directed to an artificial intelligence processing unit. The artificial intelligence processing unit comprises a plurality of identical artificial intelligence processing dies. Each of the plurality of identical artificial intelligence processing dies includes at least one inter-die input block and at least one inter-die output block. Each of the plurality of identical artificial intelligence processing dies is communicatively coupled to other of the plurality of identical artificial intelligence processing dies via one or more communication paths from the at least one inter-die output block of the artificial intelligence processing die to the at least one inter-die input block of the artificial intelligence processing die. Each of the plurality of identical artificial intelligence processing dies corresponds to at least one layer of a neural network.

[0004] In some implementations, one or more communication paths are of equal length. In some implementations, a first artificial intelligence processing die of a plurality of identical artificial intelligence processing dies is located adjacent to a second artificial intelligence processing die of a plurality of identical artificial intelligence processing dies, and the orientation of the second artificial intelligence processing die is shifted 180 degrees from the orientation of the first artificial intelligence processing die.

[0005] In some implementations, a first artificial intelligence processing die among a plurality of identical artificial intelligence processing dies is located adjacent to a second artificial intelligence processing die among the plurality of identical artificial intelligence processing dies, and the orientation of the second artificial intelligence processing die is the same as the orientation of the first artificial intelligence processing die.

[0006] In some implementations, the multiple artificial intelligence processing dies are arranged in a sequence, and at least one artificial intelligence processing die is configured to send data as input to another artificial intelligence processing die arranged in a position before the at least one artificial intelligence processing die in the sequence.

[0007] In some implementations, each artificial intelligence processing die of the multiple identical artificial intelligence processing dies is configured to receive data and perform AI calculations using the received data.

[0008] In some implementations, each artificial intelligence processing die of multiple identical artificial intelligence processing dies is configured as a systolic array and performs AI calculations using the systolic array.

[0009] In some implementations, each artificial intelligence processing die of the multiple identical artificial intelligence processing dies includes at least one host interface input block that is different from the die-to-die input block and at least one host interface output block that is different from the die-to-die output block.

[0010] In some implementations, each artificial intelligence processing die of the multiple identical artificial intelligence processing dies includes at least one multiply-accumulate unit (MAC unit).

[0011] In some implementations, each artificial intelligence processing die of the multiple identical artificial intelligence processing dies includes at least one memory.

[0012] At least one embodiment is directed to a method for processing a neural network model. The method includes receiving, at a first artificial intelligence processing die of an artificial processing unit, a first set of data associated with the network. The first artificial intelligence processing die is associated with a layer of the neural network. The method includes performing, at the first artificial intelligence processing die, a first set of AI calculations associated with the layer of the neural network associated with the first artificial intelligence processing die using the first set of data associated with the neural network. The method includes transmitting result data of the first set of AI calculations performed on the first artificial intelligence processing die to a second artificial intelligence processing die of the artificial intelligence processing unit. The second artificial intelligence processing die is associated with a different layer of the neural network from the first artificial intelligence processing die.

[0013] In some implementations, the first artificial intelligence processing die is associated with an input layer of a neural network.

[0014] In some implementations, the method includes performing, in the second artificial intelligence processing die, an AI calculation associated with a layer of a neural network associated with the second artificial intelligence processing die using result data from the calculation performed in the first artificial intelligence processing die, and transmitting the result data from the AI ​​calculation performed in the second artificial intelligence processing die as feedback to the first artificial intelligence processing die.

[0015] In some implementations, the first artificial intelligence processing die and the second artificial intelligence processing die are arranged in a sequential order, with the first artificial intelligence processing die being arranged in a position before the second artificial intelligence processing die in the sequential order.

[0016] In some implementations, the method includes performing, at a first artificial intelligence processing die, a second set of AI computations associated with a layer of a neural network associated with the first artificial intelligence processing die using result data received as feedback from a second artificial intelligence processing die and a first set of data associated with the neural network, and transmitting result data of the second set of AI computations to the second artificial intelligence processing die.

[0017] In some implementations, the second artificial intelligence processing die is associated with the output layer of the neural network.

[0018] In some implementations, the method includes: performing the first artificial intelligence processing on a second artificial intelligence processing die; and performing AI calculations related to an output layer of the neural network using result data from calculations performed on the second artificial intelligence processing die. The method includes transmitting result data from the AI ​​calculations performed on the second artificial intelligence processing die to a co-processing unit communicatively coupled to the artificial intelligence processing unit.

[0019] In some implementations, the first artificial intelligence processing die and the second artificial intelligence processing die include at least one multiply accumulate unit (MAC unit).

[0020] In some implementations, the first artificial intelligence processing die and the second artificial intelligence processing die include memory.

[0021] These and other aspects and implementations are described in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated herein by reference and constitute a part of this specification.

[0022] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For clarity, not every component may be shown in every drawing. [Brief explanation of the drawings]

[0023] [Figure 1A] FIG. 1 illustrates a system for processing neural network-related tasks according to an exemplary implementation. [Figure 1B] FIG. 2 illustrates the functional logic of the artificial intelligence processing die of the artificial intelligence processing unit according to an exemplary implementation. [Figure 1C] FIG. 2 illustrates an example layout of a systolic array of an artificial intelligence processing die according to an example implementation. [Figure 2A] FIG. 2 illustrates an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 2B] FIG. 2 illustrates an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 2C] FIG. 2 illustrates an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 2D] FIG. 2 illustrates an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 3]10 is a flowchart illustrating an example method for configuring an artificial intelligence processing die according to an example implementation. [Figure 4] 1 is a flowchart illustrating an example method for processing a neural network task based on a neural network model, according to an example implementation. [Figure 5] FIG. 1 is a block diagram illustrating a general architecture for a computer system that can be used to implement the system components and methods described and illustrated herein, according to an exemplary implementation. DETAILED DESCRIPTION OF THE INVENTION

[0024] Detailed Description The present disclosure generally relates to devices, systems, and mechanisms for processing neural network workloads. Efficient processing of neural networks utilizes custom application-specific integrated circuits (ASICs). However, designing custom ASICs presents several challenges, such as long design times and high non-recurring engineering costs. Although not limited to this, these challenges are exacerbated when ASICs are produced in low volume.

[0025] The challenges of using custom ASICs can be overcome by designing standard dies configured to process neural network tasks and interconnecting multiple such identical dies in a single ASIC chip package. The number of dies interconnected in a single chip package depends on the complexity or number of layers of the neural network being processed by the host computing device. In a package with multiple identical dies, different dies are associated with different layers of the neural network, increasing the efficiency of processing neural network-related tasks. By scaling the number of dies in a single package based on the expected frequency of performing neural network tasks, standard dies can be used across multiple products, more efficiently amortizing the cost of long design time and significant non-recurring engineering expenses.

[0026] FIG. 1A illustrates a system 100 for performing neural network computational tasks according to an exemplary implementation. The system 100 includes a main processing unit 101 and an artificial intelligence processing unit (AIPU) 102. The system 100 is housed within a host computing device (not shown). Examples of host computing devices include, but are not limited to, servers and Internet of Things (IoT) devices. The AIPU 102 is a co-processing unit of the main processing unit 101. The main processing unit 101 is communicatively coupled to the AIPU 102 via one or more communication paths, such as communication paths 104a and 104b, which are part of a communication system, such as a bus. The main processing unit 101 includes a controller 105 and a memory 107. The memory 107 stores configuration data associated with the sub-processing units of the main processing unit 101 and the co-processing units coupled to the main processing unit 101. For example, memory 107 may store configuration data associated with AIPU 102. Main processing unit controller 105 is communicatively coupled to memory 107 and configured to select configuration data from memory 107 and transmit the configuration data to co-processing units coupled to main processing unit 101 or sub-processing units of main processing unit 101. The selection and transmission of configuration data by main processing unit controller 105 is described in further detail below with reference to FIG.

[0027] The AIPU 102 is configured to process computational tasks of a neural network. The AIPU 102 includes multiple artificial intelligence processing dies (AIPDs) 103a, 103b, 103c, 103d, 103e, and 103f, collectively referred to herein as AIPDs 103. The AIPDs 103 are identical to one another. As described herein, an AIPD 103 is “identical” to another AIPD 103 if each AIPD 103 is fabricated using the same die design and the implementation of hardware units on each AIPD 103 is identical to the other AIPDs 103. Thus, in this disclosure, if the die design and hardware unit implementation of two AIPDs 103 are identical, the two AIPDs 103 can be configured to process different layers of a neural network that are still considered identical. The number of AIPDs 103 included in the AIPU 102 may vary based on the number of layers of a neural network model processed by the host computing device. For example, if the host computing device is an Internet of Things (IoT) device such as a smart thermostat, the number of layers of the neural network model being processed by the AIPU 102 of the smart thermostat is likely to be less than the number of layers of the neural network model being processed by the AIPU 102 of a host computing device in a data center, such as a server in the data center.

[0028] In a host computing device processing a simple neural network model, a single AIPD 103 can efficiently process the neural network-related tasks of the host computing device. In a host computing device processing a more complex neural network model or a neural network model having multiple layers, multiple identical AIPDs 103 may be useful for efficiently processing the neural network-related tasks. Thus, in some implementations, the AIPU 102 includes a single AIPD 103, while in other implementations, the AIPU 102 includes multiple identical AIPDs 103.

[0029] In implementations in which the AIPU 102 includes multiple identical AIPDs 103, such as that shown in Figure 1A, each identical AIPD 103 is coupled to other identical AIPDs 103. Furthermore, each AIPD 103 is associated with at least one layer of the neural network being processed by the AIPU 102. The AIPDs 103 and the arrangement of multiple identical AIPDs 103 within the AIPU 102 are described in further detail below with reference to Figures 1B, 2A, and 2B.

[0030] 1B, functional logic of an implementation of AIPD 103 is shown. To provide a clearer example, only the functional logic of AIPD 103a is shown in FIG. 1B, but because each of AIPDs 103 is identical to one another, those skilled in the art will understand that the functional logic of AIPDs 103b, 103c, 103d, 103e, and 103f is identical to the functional logic of AIPD 103a. AIPD 103a includes host interface unit 113, buffer 115, controller 117, buffer 119, computation unit 121, inter-die input blocks 109a and 109b, and inter-die output blocks 111a and 111b.

[0031] The host interface unit 113 includes at least one input / output (I / O) block (not shown). The I / O block includes multiple I / O pins (not shown). The I / O pins of the I / O blocks of the host interface unit 113 are configured bidirectionally so that the I / O blocks can receive data from a source unit and transmit data to a destination unit. Examples of source and destination units include, but are not limited to, memory units, coprocessors of the main processing unit 101, or other integrated circuit components configured to transmit and receive data. The host interface unit 113 is configured to receive data from the main processing unit controller 105 via the I / O pins of the host interface unit 113 and transmit data to the main processing unit controller 105, to the main processing unit 101, to itself, or directly to the memory 103 via the I / O pins of the host interface unit 113. The host interface unit 113 stores data received from the main processing unit controller 105 in a buffer 115.

[0032] Buffer 115 includes memory, such as a register, dynamic random access memory (DRAM), static random access memory (SRAM), or other type of integrated circuit memory, for storing data. AIPD controller 117 is configured to retrieve data from buffer 115 and store the data in buffer 115. AIPD controller 117 is configured to operate based in part on data transmitted from main processing unit controller 105. If the data transmitted from main processing unit controller 105 is configuration data, based on the configuration data, AIPD controller 117 controls an inter-die input block used for communication between AIPD 103a and other AIPDs 103. and an inter-die output block. Communication between AIPDs 103 is described in further detail below with reference to FIGS. 2A, 2B, and 2C. When the data transmitted from main processing unit controller 105 is an instruction to perform a neural network task, AIPD controller 117 is configured to store data related to the neural network in buffer 119 and perform the neural network task using the input data stored in buffer unit 119 and calculation unit 121. Buffer 119 includes memory, such as registers, DRAM, SRAM, or other types of integrated circuit memory, for storing data. Calculation unit 121 includes multiple multiply-accumulate units (MACs, not shown), multiple arithmetic logic units (ALUs) (not shown), and multiple shift registers (not shown). Some of the registers in buffer 119 are coupled to the multiple ALUs of calculation unit 121 to establish a systolic array. With a systolic array, input values ​​are read once and used for multiple different operations without storing the results before using them as inputs in subsequent operations. An example of such a systolic array arrangement is shown in Figure 1C.

[0033] In FIG. 1C , register 130 is included in buffer 119, and data from register 130 is input for a first operation on ALU 132a. The result from ALU 132a is input to ALU 132b, the result from ALU 132b is input to ALU 132c, the result from ALU 132c is input to ALU 132d, and so on. Such an arrangement and configuration distinguishes AIPD 103 from general-purpose computers, which typically store result data from one ALU in a storage unit before using the result data again. Additionally, the arrangement shown in FIG. 1C optimizes AIPD 103 for calculations related to performing artificial intelligence tasks (referred to herein as “AI calculations”), such as convolution, matrix multiplication, pooling, and element-wise vector operations. Furthermore, by implementing the arrangement shown in FIG. 1C , AIPD 103 is more optimized for power consumption and size in performing AI calculations, thereby reducing the cost of AIPU 102.

[0034] Referring again to FIG. 1B , the computation unit 121 performs AI computations using input data and weights selected for the neural network and transmitted from a weight memory unit (not shown). In some implementations, the computation unit 121 includes an activation unit 123. The activation unit 123 may include multiple ALUs and multiple shift registers and may be configured to apply activation functions and nonlinear functions to the results of the AI ​​computations. The activation functions and nonlinear functions applied by the activation unit 123 may be implemented in hardware, firmware, software, or a combination thereof. The computation unit 121 transmits the resulting data after applying the activation functions and / or nonlinear functions to the buffer 119, which stores the data. The AIPD controller 117 transmits the output data from the computation unit 121 stored in the buffer 119 to the AIPD 103 communicatively coupled to the AIPD 103a using an inter-die output block configured for inter-die communication. The inter-die communication path between two AIPDs 103 is determined by the configuration data received from the main processing unit controller 105. For example, if the configuration data received by the AIPD 103a indicates that the inter-die output block 111a (as shown in FIG. 1B) should be used for inter-die communication, the AIPD controller 117 uses the inter-die output block 111a to transmit data to the other AIPD 103. Similarly, if the configuration data indicates that the input block 109b (as shown in FIG. 1B) should be used for inter-die communication, the AIPD controller 117 selects the input block 109b as the inter-die input block for receiving data from the other AIPD 103 and reads and processes the data received by the input block 109b.

[0035] Each inter-die input block and inter-die output block of an AIPD 103 includes multiple pins. The pins of the inter-die output block of an AIPD 103 are connectable by electrical interconnections to corresponding pins of the inter-die input blocks of other AIPDs 103. For example, as shown in FIG. 2A , the pins of the output block 111a of AIPD 103a are connected by electrical interconnections to the input block of AIPD 103b. The electrical interconnections between the pins of the inter-die output blocks and inter-die input blocks of different AIPDs 103 are of equal length.

[0036] While the connection between the inter-die output block of one AIPD 103 and the inter-die input block of another AIPD 103 is made by an electrical interconnect, the selection of a particular inter-die output block of the AIPD 103 and the transmission of a particular signal or data to a particular pin of the inter-die output block are programmable or changeable based on configuration data received by the AIPD 103 from the main processing controller 105. By selecting different output blocks of the AIPD 103, the AIPU 102 can be configured to realize different requirements of different neural networks, including, but not limited to, feedback loops between different layers of the neural network. Therefore, the same AIPU 102 can be used to run various sets of neural networks, resulting in reduced design time costs and efficient amortization of non-recurring engineering expenses. The configuration of the AIPD 103 and the AIPU 102 is described in further detail below with reference to FIGS. 2A, 2B, and 3.

[0037] As described above, each AIPD 103 of the plurality of AIPDs 103 is associated with at least one layer of a neural network that the AIPU 102 is configured to process. The main processing unit 101 includes configuration data that configures the AIPDs 103 and AIPUs, such as the AIPU 102. The configuration data is associated with a neural network model selected to be processed by the AIPU. The configuration data specifies an association between the AIPD 103 and a layer of the neural network being processed by the AIPU. Based on the configuration data associated with the neural network being processed by the AIPU, the main processing unit controller 105 associates the AIPD 103 with the layer of the neural network. In some implementations, the main processing unit controller 105 stores the association between the AIPD 103 and the layer of the neural network in a storage device, such as the memory 107 (shown in FIG. 1A ). The main processing unit controller 105 transmits the configuration data associated with the AIPD 103 to the corresponding AIPD 103. The association of an AIPD 103 with a layer of a neural network is based in part on the requirements of the neural network model being processed by the AIPU 102. For example, if the neural network includes a feedback loop between two layers of the neural network, the AIPD 103 associated with those two layers can be selected in part based on whether the inter-die output block of the first AIPD 103 and the inter-die input block of the second AIPD 103 are electrically interconnected. An example of such an arrangement of multiple AIPDs 103 is described with reference to FIG. 2A.

[0038] 2A is a diagram illustrating an example of an arrangement of multiple AIPDs 103 within an AIPU, such as AIPU 102. In FIG. 2A, AIPU 102 includes six AIPDs 103 (AIPDs 103a, 103b, 103c, 103d, 103e, and 103f) processing a neural network with six layers, including a feedback loop between the last layer and the first layer of the neural network. AIPD 103a includes inter-die input blocks 109a and 109b, inter-die output blocks 111a and 111b, and a host interface unit 113. AIPD 103b includes inter-die input blocks 221a and 221b, inter-die output blocks 111a and 111b, and a host interface unit 113. AIPD 103c includes inter-die input blocks 225a and 225b, inter-die output blocks 227a and 227b, and host interface unit 215. AIPD 103d includes inter-die input blocks 229a and 229b, inter-die output blocks 231a and 231b, and host interface unit 216. AIPD 103e includes inter-die input blocks 233a and 233b, inter-die output blocks 235a and 235b, and host interface unit 217. AIPD 103f includes inter-die input blocks 237a and 237b, inter-die output blocks 239a and 239b, and host interface unit 218.

[0039] Each AIPD 103 is associated with a particular layer of the neural network, and as described above, the association of an AIPD 103 with a layer of the neural network is based in part on characteristics associated with that layer of the neural network. Because the neural network of FIG. 2A requires a feedback loop between the last layer and the first layer of the neural network, the last layer and the first layer of the neural network should be associated with an AIPD 103, where the inter-die output block of the AIPD 103 associated with the last layer of the neural network is electrically interconnected with the inter-die input block of the AIPD 103 associated with the first layer of the neural network. Because the inter-die output block 231a of AIPD 103d is electrically interconnected with the inter-die input block 109b of AIPD 103a, as shown in FIG. 2A, such an arrangement can be made by associating AIPD 103a with the first layer and AIPD 103d with the sixth layer. Thus, AIPDs 103b, 103c, 103f, and 103e are associated with the second, third, fourth, and fifth layers of the neural network, respectively. The order of placement of AIPDs 103 in Figure 2A is as follows: AIPD 103a is in the first position in the order, AIPD 103b is in the second position, AIPD 103c is in the third position, AIPD 103f is in the fourth position, AIPD 103e is in the fifth position, AIPD 103d is in the sixth position, and AIPD 103a is in the seventh position. The order of communication of neural network-related data between AIPDs 103 is shown at 201a, 201b, 201c, 201d, 201e, and 201f, starting from 103a, then to 103b, then to 103c, 103f, 103e, and 103d, and then back to 103a, incorporating a feedback layer between the sixth and first layers of the neural network. As described herein, "neural network-related data" includes, but is not limited to, calculation result data such as the output of calculation unit 121, parameter weight data, and other neural network parameter-related data.

[0040] The AIPD controller of the AIPD 103 associated with the output layer of the neural network is configured to send result data from the output layer to the main processing unit 101. For example, if the AIPD associated with the output layer is 103d, the AIPD controller 216 is configured to send result data from AIPD 103d to the main processing unit 101. In some implementations, a single AIPD 103 is configured to receive initial input data of the neural network from the main processing unit 101 and send result data from the last layer of the neural network to the main processing unit 101. For example, in FIG. 2A , if AIPD 103a receives initial input data of the neural network from the main processing unit 101 and result data from AIPD 103d, the AIPD associated with the last layer of the neural network, the AIPD controller 113 of AIPD 103a can be configured to send result data from AIPD 103d received at inter-die input block 111b to the main processing unit 101.

[0041] The same AIPD 103 described above can be used to process a neural network different from the neural network described with reference to FIG. 2A. For example, if the neural network has a feedback loop between the sixth layer and the third layer of the neural network, the sixth layer and the third layer should be associated with an AIPD 103, in which the inter-die output block of the AIPD 103 associated with the sixth layer of the neural network is electrically interconnected with the inter-die input block of the AIPD 103 associated with the third layer of the neural network. Furthermore, each of the AIPDs 103 associated with different layers of the neural network is electrically interconnected with at least one inter-die input block of another AIPD 103 associated with a subsequent layer of the neural network. For example, an AIPD 103 associated with a first layer should have an inter-die output block electrically interconnected with an inter-die input block of an AIPD 103 associated with a second layer of the neural network, an AIPD 103 associated with a second layer should have an inter-die output block electrically interconnected with an inter-die input block of an AIPD 103 associated with a third layer of the neural network, an AIPD 103 associated with a third layer should have an inter-die output block electrically interconnected with an inter-die input block of an AIPD 103 associated with a fourth layer of the neural network, an AIPD 103 associated with a fourth layer should have an inter-die output block electrically interconnected with an inter-die input block of an AIPD 103 associated with a fifth layer of the neural network, and an AIPD 103 associated with a fifth layer should have an inter-die output block electrically interconnected with an inter-die input block of an AIPD 103 associated with a sixth layer of the neural network. Such neural network processing can be performed using the AIPD 103 arrangement shown in FIG. 2B.

[0042] 2B is a diagram showing a different example of the arrangement of AIPDs 103 within an AIPU. In FIG. 2B, an AIPU 250 includes AIPDs 103a, 103b, 103c, 103d, 103e, and 103f. Within AIPU 250, inter-die output block 111a of AIPD 103a is electrically interconnected with inter-die input block 221a of AIPD 103b, inter-die output block 111b of AIPD 103a is electrically interconnected with inter-die input block 229a of AIPD 103d, inter-die output block 223b of AIPD 103b is electrically interconnected with inter-die input block 109b of AIPD 103a, inter-die output block 223a of AIPD 103b is electrically interconnected with inter-die input block 225a of AIPD 103c, and inter-die output block 227b of AIPD 103c is electrically interconnected with inter-die input block 237a of AIPD 103f. The die-to-die output block 239a of AIPD103f is electrically connected to the die-to-die input block 225b of AIPD103c, the die-to-die output block 239b of AIPD103f is electrically interconnected to the die-to-die input block 233b of AIPD103e, the die-to-die output block 235a of AIPD103e is electrically interconnected to the die-to-die input block 221b of AIPD103b, the die-to-die output block 235b of AIPD103e is electrically interconnected to the die-to-die input block 229b of AIPD103d, and the die-to-die output block 231a of AIPD103d is electrically interconnected to the die-to-die input block 233a of AIPD103e.

[0043] 2B, AIPD 103f is associated with the sixth layer of the neural network, and AIPD 103e is associated with the third layer of the neural network. AIPDs 103a, 103d, 103b, and 103c are associated with the first, second, fourth, and fifth layers of the neural network, respectively. AIPD controller 113 transmits the result data of the calculation in AIPD 103a to AIPD 103d, the AIPD 103 associated with the second layer of the neural network, via an inter-die output block 229a of AIPD 103a electrically interconnected to an inter-die input block 229a of AIPD 103d. The AIPD controller 216 of AIPD 103d is configured to transmit the resulting data from AIPD 103d to AIPD 103e, the AIPD 103 associated with the third layer of the neural network, using inter-die output block 231a electrically interconnected to inter-die input block 233a of AIPD 103e. The AIPD controller 217 of AIPD 103e is configured to transmit the resulting data from AIPD 103e to AIPD 103b, the AIPD 103 associated with the fourth layer of the neural network, using inter-die output block 235a of AIPD 103e electrically interconnected to inter-die input block 221b of AIPD 103b. The AIPD controller 214 of AIPD 103b is configured to transmit the resulting data from AIPD 103b to AIPD 103c, the AIPD 103 associated with the fifth layer of the neural network, using the inter-die output block 223a electrically interconnected to the inter-die input block 225a of AIPD 103c. The AIPD controller 215 of AIPD 103c is configured to transmit the resulting data from AIPD 103c to AIPD 103f, the AIPD 103 associated with the sixth layer of the neural network, using the inter-die output block 227b of AIPD 103c electrically interconnected to the inter-die input block 237a of AIPD 103f. The AIPD controller 218 is configured to send feedback data from the AIPD 103f to the AIPD 103e, the AIPD 103 associated with the third layer of the neural network, using the inter-die output block 239b of the AIPD 103f, which is electrically interconnected to the inter-die input block 233b of the AIPD 103e. The AIPD controller 218 of the AIPD 103f is further configured to send result data from the AIPD 103f to the main processing unit 101 when the AIPD 103f is associated with the output layer of the neural network.The order of placement of the AIPDs 103 in Figure 2B is such that AIPD 103a is in the first position in the order, AIPD 103d is in the second position, AIPD 103e is in the third position, AIPD 103b is in the fourth position, AIPD 103c is in the fifth position, AIPD 103f is in the sixth position, and AIPD 103e is in the seventh position. The order of communication of neural network-related data between AIPDs 103 in Figure 2B starts with 103a, then 103d, 103e, 103b, 103c, 103f, and then sends feedback data to 103e, as shown by 202a, 202b, 202c, 202d, 202e, and 202f.

[0044] Therefore, the same AIPD can be used to process different neural networks with different neural network requirements. Thus, a single artificial intelligence processing die (AIPD) design can be used to process and execute different neural networks with different requirements, thereby reducing costs associated with design time and efficiently amortizing non-recurring engineering costs.

[0045] Additionally, a single AIPU can be used to process different neural networks by modifying the configuration data associated with the AIPU and / or the configuration data associated with the AIPDs of the AIPU. For example, in FIG. 2B , if a neural network having four layers is processed by AIPU 250, the configuration data associated with AIPU 250 and / or the configuration data associated with AIPDs 103 of AIPU 250 can be modified to associate AIPD 103a with the first layer of the neural network, AIPD 103b with the second layer of the neural network, AIPD 103c with the third layer of the neural network, and AIPD 103f with the fourth layer of the neural network. The electrical interconnections between the inter-die output blocks and inter-die input blocks of these AIPDs 103 are described above. When AIPU 250 and the AIPDs 103 of AIPU 250 are reconfigured, the main processing unit The inter-die controller 105 transmits input data associated with the neural network to an AIPD, AIPD 103a, associated with the first layer of the neural network. Based on the modified configuration data associated with AIPD 103a and the input data to the neural network, AIPD 103a performs calculations, including AI calculations, associated with the first layer of the new neural network and transmits the resulting data to AIPD 103b using inter-die output block 111a. As described herein, "computations associated with a layer of a neural network" includes AI calculations associated with that layer of the neural network. AIPD 103b performs calculations, including AI calculations, associated with the second layer of the neural network based on the resulting data received from AIPD 103 at inter-die input block 221a and the modified configuration data associated with AIPD 103b. AIPD 103b transmits the resulting data to AIPD 103c using inter-die output block 223a. AIPD 103c performs calculations associated with the third layer of the neural network, including AI calculations, based on the result data received from AIPD 103b via inter-die input block 225a and the modified configuration data associated with AIPD 103c, and transmits the result data to AIPD 103f via inter-die output block 227b. AIPD 103f performs calculations associated with the fourth layer of the neural network, including AI calculations, based on the result data received from AIPD 103c via inter-die input block 237a and the modified configuration data associated with AIPD 103f. AIPD 103f, the AIPD 103 associated with the last layer of the neural network, is configured to transmit the result data from AIPD 103f to main processing unit 101. Thus, a single AIPU can be reprogrammed to process different neural networks by modifying the configuration data associated with the AIPU and / or the configuration data of the AIPD of the AIPU.Therefore, the significant non-recurring engineering costs associated with using a custom ASIC can be more effectively amortized, further reducing the design time costs associated with designing a custom ASIC to handle this particular neural network task.

[0046] In some implementations, at least one inter-die input block and at least one inter-die output block are located on one edge of the AIPD 103, and at least one inter-die output block and at least one inter-die input block are provided on the other edge of the AIPD 103. For example, as shown in FIG. 2A , one inter-die input block and one inter-die output block are provided on the top edge of the AIPD 103, and the other inter-die output block and inter-die input block are provided on the bottom edge of the AIPD 103. In some implementations, all inter-die input blocks are provided on one edge of the AIPD 103, and all inter-die output blocks are provided on the other edge of the AIPD 103b. An example of such an arrangement of inter-die input blocks and inter-die output blocks is shown in FIG. 2C.

[0047] 2C , all inter-die input blocks are located on the top edge of the AIPD 103, and all inter-die output blocks are located on the bottom edge of the AIPD 103. In some implementations, to achieve equal length electrical interconnections between the AIPDs 103 and to achieve a more efficient size for an AIPU that includes the AIPD 103 shown in FIG. 2C , the orientation of some of the AIPDs 103 is shifted by a specific distance or angle relative to the orientation of other AIPDs 103. For example, as shown in FIG. 2C , AIPDs 103b and 103e are rotated 180 degrees relative to the orientation of AIPDs 103a, 103d, 103c, and 103f. By rotating the AIPDs 103 by 180 degrees, the die-to-die input and die-to-die output blocks of the AIPDs 103b and 103e are located adjacent to the die-to-die output and die-to-die input blocks of the AIPDs 103a, 103d, 103c, and 103f, thereby making the lengths of the electrical interconnections between all of the AIPDs 103 equal and reducing the inter-die interconnections between the AIPDs 103b or 103e and any adjacent AIPDs 103. No additional area is required for electrical interconnections between the input block and the inter-die output block.

[0048] 2C, an AIPU having the arrangement of AIPDs 103 shown in FIG. 2C can process neural networks similar to the AIPUs described above. For example, a neural network having six layers and no feedback loops between the layers can be processed by the arrangement of AIPDs 103 shown in FIG. 2C by associating AIPD 103a with the first layer of the neural network, AIPD 103d with the second layer, AIPD 103e with the third layer, AIPD 103b with the fourth layer, AIPD 103c with the fifth layer, and AIPD 103f with the sixth layer. The order of the arrangement of AIPDs 103 in FIG. 2C is as follows: AIPD 103a is in the first position in the order, AIPD 103d is in the second position, AIPD 103e is in the third position, AIPD 103b is in the fourth position, AIPD 103c is in the fifth position, and AIPD 103f is in the sixth position. Communication between the AIPDs 103 begins with AIPD 103a, then goes to AIPDs 103d, 103e, 103b, 103c, and 103f in that order.

[0049] One of the advantages of the AIPD design and implementation described herein is that any number of AIPDs can be included within a single AIPU package. The number of AIPDs within a single AIPU package is limited only by the size of the AIPU package, not by the size of the AIPD die. Thus, as shown by the arrangement of AIPDs 11 through NN in FIG. 2D, it is possible to include an N×N arrangement of AIPDs in a single AIPU package. AIPDs 11 through NN in FIG. 2D are designed and configured similarly to AIPD 103 described above.

[0050] The main processing unit controller 105 is configured to send initial input data of the neural network to the AIP 103 associated with the first layer (input layer) via the host interface unit of the AIPD 103. For example, as shown in FIGS. 2, 2B, and 2C, the AIPD 103 associated with the first layer of the neural network is AIPD 103a, and the main processing unit controller 105 sends the initial input data to AIPD 103a via the host interface unit 113. In some implementations, the last AIPD 103 in the communication order is configured to send result data back to the main processing unit 101 using the AIPD's host interface unit. In some implementations, the AIPD 103 associated with the last layer of the neural network is configured to send result data back to the main processing unit 101. 2A, as described above, the last AIPD 103 in the communication order is AIPD 103a, and therefore, in some implementations, AIPD controller 117 of AIPD 103a is configured to send the result data to main processing unit 101 using host interface unit 113. Similarly, in FIG. 2B, AIPD 103f is the AIPD 103 associated with the last layer of the neural network, and in some implementations, AIPD controller 218 of AIPD 103f is configured to send the result data to main processing unit 101 using AIPD 103f's host interface unit. An example of how to configure AIPD 103 for neural network processing is described with reference to FIG. 3.

[0051] FIG. 3 is a flow chart illustrating an example method 300 for configuring an AIPU to process a neural network model. In a main processor, the method 300 comprises receiving input to configure the AIPU (stage 302). The method 300 comprises selecting AIPU configuration data (stage 304). The method 300 comprises transmitting the configuration data to the AIPD 103 of the AIPU (stage 305). At each AIPD 103, the method 300 comprises receiving configuration data (stage 308). The method 300 comprises configuring the AIPD 103 based on the configuration data (stage 310). The method 300 comprises sending a response to the main processing unit 101 (stage 312).

[0052] The method 300 comprises receiving input in the main processing unit 101 to configure the AIPU (stage 302). In response to receiving the input to configure the AIPU, the method 300 comprises selecting AIPU configuration data for each AIPD 103 in the AIPU (stage 304). The main processing unit controller 105 of the main processor 101 selects configuration data associated with the AIPU. In selecting the configuration data associated with the AIPU, the main processing unit controller 105 selects configuration data associated with each of the AIPDs 103 of the AIPU. Different configuration data may specify different values ​​for configuring the AIPDs 103 for neural network processing. The AIPDs 103 include, but are not limited to, inter-die output blocks and inter-die input blocks of the associated AIPDs 103, which are configured for sending and receiving neural network-related data between the associated AIPD 103 and other AIPDs 103, and for mapping output data to pins of the inter-die output blocks 103 and neural network-related data such as parameters, parameter weight data, and number of parameters. The values ​​specified by the configuration data are based on the layer of the neural network with which the corresponding AIPD 103 is associated. Thus, the value of the configuration data associated with one AIPD 103 may be different from the value of the configuration data associated with a different AIPD 103.For example, if a first layer of a neural network being processed by an AIPU requires a first set of weight values ​​to be used for a computational task of the first layer of the neural network, and a second layer of the neural network requires a second set of weight values ​​different from the first set of weight values ​​to be applied during a computational task of the second layer, then the configuration data associated with the AIPD 103 associated with the first layer of the neural network specifies weight values ​​corresponding to the first set of weight values, while the configuration data associated with the AIPD 103 associated with the second layer of the neural network specifies weight values ​​corresponding to the second set of weight values.

[0053] The inter-die output block of the AIPD 103 specified in the configuration data for transmitting neural network-related data to the AIPD 103 associated with the next layer of the neural network is based in part on the location of the AIPD 103 relative to the AIPD 103 associated with the next layer of the neural network. For example, if AIPD 103a is associated with the first layer of the neural network and AIPD 103b is associated with the next layer of the neural network, the inter-die output block of AIPD 103a specified in the configuration data for AIPD 103a is the inter-die output block electrically interconnected to the inter-die input block of AIPD 103b, which is inter-die output block 111a, as shown in FIGS. 2A, 2B, and 2C. Similarly, when AIPD103d is associated with the layer next to the layer associated with AIPD103a, the inter-die output block selected to transmit neural network-related data and specified in the configuration data of AIPD103a is the inter-die output block electrically interconnected to the inter-die input block of AIPD103d, which is inter-die output block 111b, as shown in Figures 2A, 2B, and 2C.

[0054] Each AIPD 103 is associated with a unique identifier, and in some implementations, the configuration data of an AIPD 103 is associated with the unique identifier of that AIPD 103, and the main processing unit controller 105 is configured to select the configuration data of the AIPD 103 based on the unique identifier associated with the AIPD 103.

[0055] Method 300 includes transmitting the selected configuration data to an AIPD 103 (stage 306). As described above, the main processing unit controller 105 transmits the configuration data to the AIPD 103 via the AIPD's 103's host interface unit, such as the host interface unit 113 of the AIPD 103a. In some implementations, the main processing unit controller 105 is configured to periodically check whether the configuration data for any AIPD 103 has been updated, and in response to updating the AIPD's 103's configuration data, the main processing unit controller 105 transmits the updated configuration data to the particular AIPD 103. In some implementations, the main processing unit controller 105 transmits instructions to the AIPD 103 to configure the AIPD 103 based on the received configuration data. In some implementations, the configuration data is stored in a host computing device memory, and the AIPD 103 is configured to read the data stored in the host computing device's memory. In some implementations, the main processing unit controller 105 sends instructions to the AIPD 103 to read configuration data from the host computing device memory and configure the AIPD 103 based on the configuration data.

[0056] The method 300 includes, at each AIPD 103, receiving configuration data (stage 308) and configuring the AIPD 103 based on the received configuration data (stage 310). As described above, an AIPD controller of an AIPD 103, such as the AIPD controller 117 of the AIPD 103a, is configured to select an inter-die input block and an inter-die output block and configure these blocks to send and receive data from other AIPDs 103 based on the received configuration data. The AIPD controller of the AIPD 103 is also configured to send specific output data of the AIPD 103, such as output from the computation unit 121, to specific pins of the selected inter-die output block that are selected for transmitting the neural-network-related data to other AIPDs 103 based on the received configuration data. The AIPD controller of the AIPD 103 is further configured to store neural-network-related data, such as parameter weight data, in a storage device, such as the buffer 119, and to use the neural-network-related data during computations related to layers of the neural network associated with the AIPD 103.

[0057] The method 300 includes, at each AIPD 103, transmitting a response signal to the main processor 101 (stage 312). The AIPDs 103 transmit the response signal to the main processor 101 using a host interface unit, such as the host interface unit 113 of the AIPD 103a. The response transmitted to the main processor 101 informs the main processor that the configuration of the AIPD 103 is successful. In some implementations, if an error occurs during the configuration of the AIPD 103, the AIPD 103 transmits an error message to the main processor 101 using the host interface unit. Once the required AIPDs 103 are successfully configured, the AIPU is ready to process neural network-related tasks. The main processing unit controller 105 assigns the neural network task to the AIPU to perform the neural network task. An example of how a neural network task may be processed by an AIPU is described below with reference to FIG.

[0058] 4 is a flow chart illustrating an example method 400 for processing neural network-related tasks by an AIPU. In the main processor 101, the method 400 comprises identifying a neural network task (stage 402). The method 400 comprises transmitting initial data, or input data associated with the neural network, to the AIPU (stage 404). In the AIPU, the method 400 comprises receiving initial data associated with the neural network at a first AIPD 103 associated with an input layer of the neural network (stage 406). The method 400 comprises performing, in the first AIPD 103, a calculation associated with the layer of the neural network associated with the first AIPD 103 using the initial data and any neural network-related data received with the configuration data of the first AIPD 103 (stage 408). The method 400 comprises transmitting results of the calculation to a second AIPD (stage 410). The method 400 comprises performing, at the second AIPD 103, calculations related to a layer of a neural network associated with the second AIPD 103 using the result data received from the first AIPD (stage 412). In some implementations, the method 400 comprises transmitting results of the calculations at the second AIPD 103 as feedback to the first AIPD 103 (stage 414). The method 400 comprises transmitting the neural network results from the AIPU to the main processor (stage 416). The method 400 comprises transmitting, at the main processor, the neural network results to a user (stage 418).

[0059] The method 400 comprises identifying a neural network task (stage 402) in the main processor 101. The main processing unit controller 105 is configured to identify whether the requested task is a neural network-related task. In some implementations, the request message or data for the requested task carries a specific indicator, such as a high bit or a low bit in a specific field of the message, that indicates that the requested task is a neural network-related task, and the main processing unit controller 105 is configured to determine whether the requested task is a neural network task based on the specific indicator.

[0060] The method 400 comprises, in the main processor 101, sending input data of the neural network to the AIPU (stage 404). The main processing unit controller 105 of the main processing unit 101 retrieves the input data from the memory of the host computing device and sends it to the AIPDs 103 associated with the initial or input layer of the neural network being processed by the AIPU. The main processing unit controller 105 identifies the AIPDs 103 associated with the input layer of the neural network based on configuration data associated with each of the AIPDs 103. In some implementations, identifiers of the AIPDs 103 associated with the input layer of the neural network are stored in a storage unit such as a memory or register or buffer, and the main processing unit controller 105 determines the AIPDs 103 associated with the input layer based on the identifiers stored in the memory or storage unit. In an implementation in which the AIPD 103 is configured to read data stored in the memory of the host computing device, the main processing unit controller 105 sends instructions to the AIPD 103 associated with the input layer of the neural network to retrieve input data to the neural network from the memory of the host computing device.

[0061] The method 400 comprises receiving input data related to a neural network at a first AIPD 103 associated with an input layer of the neural network, such as the AIPD 103a described with reference to Figures 2A, 2B, and 2C (stage 406). The method 400 also comprises performing, at the first AIPD 103, calculations related to the layer of the neural network associated with the first AIPD 103 using the initial data received at the first AIPD 103 and other neural-network-related data received during configuration of the first AIPD 103 (stage 408). A controller of the first AIPD 103 determines the calculations to be performed based on the associated neural network layer. For example, if the first layer of the neural network performs matrix multiplication by applying a weight matrix to the input data, during configuration of the AIPD 103, the weight matrix is ​​transmitted to the first AIPD 103 and stored in a buffer of the AIPD 103. The AIPD controller of the first AIPD 103 is configured to send the weight matrix to a computation unit of the first AIPD 103 to perform a matrix multiplication using the weight matrix and the input data. In some implementations, the computation to be performed is specified in configuration data received by the first AIPD 103, and based on the specified computation, the controller of the first AIPD 103 sends the data to an appropriate computation unit of the AIPD 103, such as computation unit 121 of AIPD 103a.

[0062] The method 400 comprises transmitting, at a first AIPD 103, results of a computation at the first AIPD 103 to a second AIPD 103 (stage 410). The second AIPD 103 is associated with a different layer of a neural network than the first AIPD. The method 400 also comprises performing, at the second AIPD 103, a computation related to the layer of the neural network associated with the second AIPD 103 using the results data received from the first AIPD 103 and other neural network-related data (stage 412). In some implementations, a controller of the AIPD 103 performing the computation can retrieve additional data for the computation, such as parameter weight data for use in the AI ​​computation, from a memory of a host computing device.

[0063] In an implementation where the neural network model being processed by the AIPU includes a feedback loop between two or more layers of the neural network, and the second AIPD 103 and the first AIPD 103 are associated with layers of the neural network that include a feedback loop therebetween, the method 400 comprises, in the second AIPD 103, transmitting, as feedback to the first AIPD 103, data resulting from the calculation in the second AIPD 103 (stage 414). If there is no feedback loop between the layers associated with the second AIPD 103 and the first AIPD 103, the method 400 comprises transmitting, as feedback to the main processing unit 101, the neural network results from the AIPU (stage 416). The controller of the AIPD 103 associated with the output layer of the neural network transmits the neural network results to the main processing unit 101 using a host interface, such as the host interface unit 113 of the AIPD 103a. For example, in Figure 2A, AIPD 103a is the AIPD 103 associated with the output layer of the neural network in Figure 2A, and therefore, the AIPD controller 117 of AIPD 103a sends the result data to the main processor 101 using the host interface unit 113. Similarly, in Figure 2B, AIPD 103f is associated with the output layer of the neural network in Figure 2B, and the AIPD controller of AIPD 103f sends the result data to the main processor 101 using the host interface unit of AIPD 103f.

[0064] The method 400 includes, in the main processing unit 101, transmitting the neural network results received from the AIPU to the requester of the neural network task. (stage 418). As used herein, a "neural network task requestor" may be another process within the host computing device or an end user of the host computing device. While only two AIPDs 103 are shown in FIG. 4 for clarity and to provide a clear example, the number of AIPDs 103 used in performing a neural network task will depend, at least in part, on the volume of neural network tasks expected to be performed by the host computing device.

[0065] 5 is a block diagram illustrating an overall architecture for a computer system 500 that may be used to implement the system elements and methods described and illustrated herein, according to an exemplary implementation. The computing system 500 may be used to implement the host computing device described above. The computing system 500 may be used to implement the AIPU method 300 configuration shown in FIGS. 3 and 4 and to process neural network tasks using the AIPU method 400.

[0066] In broader overview, computing system 510 includes at least one processor 550 that performs operations according to instructions and one or more memory devices 570 or 575 that store instructions and data. The illustrated example of computing system 510 includes one or more processors 550 connected via a bus 515 to at least one network interface controller 520 having one or more network interface ports 522 connected to a network (not shown), an AIPU 590, memory 570, and other elements 580, such as an input / output (I / O) interface 530. Generally, processor 550 executes instructions received from memory. The illustrated processor 550 incorporates or is directly connected to a cache memory 575.

[0067] More specifically, processor 550 may be any logic circuitry that processes instructions, for example, instructions retrieved from memory 570 or cache 575. In many embodiments, processor 550 is a microprocessor unit or a special-purpose processor. Computing device 500 may be any processor or collection of processors capable of operating as described herein. In some implementations, processor 550 is capable of performing particular stages of method 300 shown in FIG. 3 , such as stages 302, 304, and 306, and particular stages of method 400 shown in FIG. 4 , such as stages 402, 404, and 418. Processor 550 may be a single-core or multi-core processor. Processor 550 may be multiple processors. In some implementations, processor 550 is configurable to perform multithreaded operations. In some implementations, processor 550 may host one or more virtual machines or containers in conjunction with a hypervisor or container manager for managing the operation of these virtual machines or containers. In such an implementation, the method 300 shown in FIG. 3 and the method 400 shown in FIG. 4 may be implemented within a virtual or container environment provided on the processor 550.

[0068] The memory 570 may be any device suitable for storing computer-readable data. The memory 570 may be a device having a fixed storage device or a device for reading a removable storage medium. Examples include all forms of non-volatile memory, media, and memory devices, semiconductor memory devices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magnetic disks, magneto-optical disks, and optical disks (e.g., CD-ROM, DVD-ROM, and Blu-ray discs). The computing system 500 may have any number of memory devices 570. In some implementations, the memory 570 may be configured to store the data in accordance with the method 300 shown in FIG. 3 and the method 300 shown in FIG. 4. In some implementations, memory 570 supports virtual memory or container memory accessible in a virtual machine or container execution environment provided by computing system 510.

[0069] Cache memory 575 is typically a form of computer memory located in close proximity to processor 550 for fast read times. In some implementations, cache memory 575 is part of processor 550 or on the same chip as processor 550. In some implementations, there are multiple levels of cache 575, e.g., L2, L3 cache layers.

[0070] The network interface controller 520 manages data exchange through the network interface 522 (also referred to as a network interface port). The network interface controller 520 handles the physical and data link layers of the OSI model for network communication. In some implementations, some of the network interface controller's tasks are handled by the processor 550. In some implementations, the network interface controller 520 is part of the processor 550. In some implementations, the computing system 510 has multiple network interface controllers 520. The network interface 522 is a connection point for a physical network link. In some implementations, the network interface controller 520 supports wireless network connections, and the interface port 522 is a wireless receiver / transmitter. Generally, the computing device 510 exchanges data with other computing devices through a physical or wireless link to the network interface 522. The network interface 522 may link directly to other devices or may be connected through an intermediate device, such as a hub, bridge, switch, or router, that connects the computing device 510 to a network, such as the Internet. In some implementations, the network interface controller 520 implements a network protocol such as Ethernet.

[0071] Other elements 580 may include I / O interface 530, external serial device ports, and additional coprocessors. For example, computing system 510 may include interfaces (e.g., universal serial bus (USB) interfaces) for connecting input devices (e.g., keyboard, microphone, mouse, or other pointing device), output devices (e.g., video display, speakers, or printer), or additional memory devices (e.g., portable flash drives or external media drives). In some implementations, other components 580 include additional coprocessors, such as math coprocessors, that assist processor 550 in high-precision or complex calculations.

[0072] Implementations of the inventive subject matter and operations described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware embodied on a tangible medium, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Implementations of the inventive subject matter described herein can also be implemented as one or more computer programs embodied on a tangible medium, i.e., as one or more modules of computer program instructions encoded on one or more computer storage media for execution by or to control the operation of a data processing apparatus. A computer storage medium may be a computer-readable memory device, a computer-readable storage substrate, a random or serial access memory array or element, or a combination of one or more of these. The computer storage medium may be or be included in a combination of one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The computer storage medium may be a tangible, non-transitory computer storage medium.

[0073] The operations described herein may be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources. The operations may be performed within the native environment of the data processing device or within one or more virtual machines or containers hosted by the data processing device.

[0074] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored among files that hold other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple cooperating files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to run on a single computer or in one or more virtual machines or containers located at one location or distributed across multiple locations and interconnected by a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0075] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0076] While this specification contains many specific implementation details, these should not be construed as limitations on the invention or the claims, but rather as descriptions of features specific to particular implementations of a particular invention. Some features described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented in separate implementations or in any suitable subcombination. Also, while features may be described above as operating in several combinations and initially claimed as such, one or more features from a claimed combination can, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0077] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown to achieve desirable results, or as requiring all of the operations shown to be performed to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, It should be understood that the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0078] The reference of "or" may be construed as inclusive, such that any term listed using "or" refers to either one, more than one, and all of the listed terms. The designations "first," "second," "third," etc. are not necessarily intended to indicate an ordering, but are merely used to generally distinguish between like or similar items or elements.

[0079] Various modifications to the implementations described herein will be apparent to those skilled in the art, and the general principles described herein can be applied to other implementations without departing from the spirit or scope of the disclosure. Accordingly, the scope of the claims is not intended to be limited to the implementations described herein, but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. an artificial intelligence processing unit comprising a plurality of identical artificial intelligence processing dies; each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies includes at least one inter-die input block and at least one inter-die output block; each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies is communicatively coupled to another artificial intelligence processing die of the plurality of identical artificial intelligence processing dies via one or more communication paths from the at least one inter-die output block of the artificial intelligence processing die to the at least one inter-die input block of the artificial intelligence processing die; An artificial intelligence processing unit, wherein each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies corresponds to at least one layer of a neural network.

2. The artificial intelligence processing unit of claim 1 , wherein the one or more communication paths are of equal length.

3. 2. The artificial intelligence processing unit of claim 1, wherein a first artificial intelligence processing die among the plurality of identical artificial intelligence processing dies is located adjacent to a second artificial intelligence processing die among the plurality of identical artificial intelligence processing dies, and the orientation of the second artificial intelligence processing die is shifted 180 degrees from the orientation of the first artificial intelligence processing die.

4. 2. The artificial intelligence processing unit of claim 1, wherein a first artificial intelligence processing die among the plurality of identical artificial intelligence processing dies is located adjacent to a second artificial intelligence processing die among the plurality of identical artificial intelligence processing dies, and the orientation of the second artificial intelligence processing die is the same as the orientation of the first artificial intelligence processing die.

5. 2. The artificial intelligence processing unit of claim 1, wherein the plurality of artificial intelligence processing dies are arranged in a sequence, and at least one artificial intelligence processing die is configured to send data as input to another artificial intelligence processing die arranged in a position before the at least one artificial intelligence processing die in the sequence.

6. 2. The artificial intelligence processing unit of claim 1, wherein each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies is configured to receive data and perform AI calculations using the received data.

7. The artificial intelligence processing unit of claim 6 , wherein each of the plurality of identical artificial intelligence processing dies is configured as a systolic array, and the AI ​​calculation is performed using the systolic array.

8. 2. The artificial intelligence processing unit of claim 1, wherein each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies includes at least one host interface input block different from the die-to-die input block and at least one host interface output block different from the die-to-die output block.

9. 9. The artificial intelligence processing unit of claim 8, wherein each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies is configured to receive data from a co-processing unit via the host interface input block and to transmit data to the co-processing unit via the host interface output block.

10. The artificial intelligence processing unit of claim 1 , wherein each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies includes at least one multiply-accumulate unit (MAC unit).

11. The artificial intelligence processing unit of claim 1 , wherein each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies includes at least a memory.

12. 1. A method comprising: receiving a first set of data associated with a neural network at a first artificial intelligence processing die of an artificial intelligence processing unit, the first artificial intelligence processing die being associated with a layer of the neural network; performing, in the first artificial intelligence processing die, a first set of AI calculations associated with the layer of the neural network associated with the first artificial intelligence processing die using the first set of data associated with the neural network; and transmitting result data of the first set of AI calculations performed in the first artificial intelligence processing die to a second artificial intelligence processing die of the artificial intelligence processing unit, the second artificial intelligence processing die being associated with a different layer of the neural network from the first artificial intelligence processing die.

13. The method of claim 12 , wherein the first artificial intelligence processing die is associated with the input layer of the neural network.

14. performing, in the second artificial intelligence processing die, AI calculations related to the layer of the neural network associated with the second artificial intelligence processing die using the result data of the calculations performed in the first artificial intelligence processing die; 13. The method of claim 12, further comprising: transmitting result data of the AI ​​calculations performed in the second artificial intelligence processing die as feedback to the first artificial intelligence processing die.

15. 15. The method of claim 14, wherein the first artificial intelligence processing die and the second artificial intelligence processing die are arranged in order, and the first artificial intelligence processing die is arranged in a position before the second artificial intelligence processing die in the order.

16. performing, in the first artificial intelligence processing die, a second set of AI calculations associated with the layer of the neural network associated with the first artificial intelligence processing die using the result data received as feedback from the second artificial intelligence processing die and the first set of data associated with the neural network; 15. The method of claim 14, further comprising: transmitting result data of the second set of AI calculations to the second artificial intelligence processing die.

17. The method of claim 12 , wherein the second artificial intelligence processing die is associated with the output layer of the neural network.

18. In the second artificial intelligence processing die, performing AI calculations related to the output layer of the neural network using the result data of the calculations performed in the first artificial intelligence processing die; 17. The method of claim 16, further comprising: transmitting result data of the AI ​​calculations performed in the second artificial intelligence processing die to a co-processing unit communicatively coupled to the artificial intelligence processing unit.

19. 13. The method of claim 12, wherein the first artificial intelligence processing die and the second artificial intelligence processing die include at least one multiply-accumulate unit (MAC unit).

20. The method of claim 12 , wherein the first artificial intelligence processing die and the second artificial intelligence processing die include memory.

Images