Apparatus and mechanism for processing neural network tasks using a single chip package having multiple identical dies

Abstract

To provide an artificial intelligence processing unit for and a method of processing a neural network model.SOLUTION: An artificial intelligence processing unit has a plurality of identical artificial intelligence processing dies 103a to 103f. For example, the artificial intelligence processing die 103a has at least one inter-die input block 109a, 109b, and at least one inter-die output block 111a, 111b. In each of the artificial intelligence processing dies, dies from the at least one inter-die output block (e.g., the inter-die output block 111a) of the artificial intelligence processing die (e.g., artificial intelligence processing die 103a) to at least one inter-die input block (e.g., inter-die input block 221a) of the artificial intelligence processing die (e.g., the artificial intelligence processing die 103b) are connected to one another through one or more communication paths so as to be capable of communicating with one another. Each of the artificial intelligence processing dies corresponds to at least one layer of a neural network.SELECTED DRAWING: Figure 2A

Description

Background Art

[0001] Related Applications This application claims priority and benefit under 35 U.S.C. § 119(e) to U.S. Patent Application No. 15 / 819,753, filed on Nov. 21, 2017, entitled “APPARATUS AND MECHANISM FOR PROCESSING NEURAL NETWORK TASKS USING A SINGLE CHIP PACKAGE WITH MULTIPLE IDENTICAL DIES,” the entire contents of which are incorporated herein by reference for all purposes.

[0002] Background The use of neural networks in the field of artificial intelligence computing has grown rapidly over the past several years. More recently, the use of special-purpose computers, such as application-specific integrated circuits (ASICs), has been used to process neural networks. However, the use of ASICs presents several challenges. Some of these challenges are: (1) long design times and (2) non-negligible non-recurring engineering costs. As the popularity of neural networks grows and the scope of tasks regarding which neural networks are used expands, the long design times and non-negligible non-recurring engineering costs worsen.

Summary of the Invention

[0003] Summary At least one embodiment is directed towards an artificial intelligence processing unit. The artificial intelligence processing unit comprises a plurality of identical artificial intelligence processing dies. Each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies includes at least one die-to-die input block and at least one die-to-die output block. Each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies is communicably connected to other artificial intelligence processing dies of the plurality of identical artificial intelligence processing dies via one or more communication paths from at least one die-to-die output block of the artificial intelligence processing die to at least one die-to-die input block of the artificial intelligence processing die. Each artificial intelligence processing die of the plurality of identical artificial intelligence processing dies corresponds to at least one layer of a neural network.

[0004] In some implementations, one or more communication channels are of equal length. In some implementations, the first artificial intelligence processing die among multiple identical artificial intelligence processing dies is located adjacent to the second artificial intelligence processing die among multiple identical artificial intelligence processing dies, and the orientation of the second artificial intelligence processing die is rotated 180 degrees from the orientation of the first artificial intelligence processing die.

[0005] In some implementations, the first artificial intelligence processing die among multiple identical artificial intelligence processing dies is located adjacent to the second artificial intelligence processing die among multiple identical artificial intelligence processing dies, and the orientation of the second artificial intelligence processing die is the same as the orientation of the first artificial intelligence processing die.

[0006] In some implementations, multiple artificial intelligence processing dies are arranged in sequence, and at least one artificial intelligence processing die is configured to send data as input to other artificial intelligence processing dies that are positioned earlier in the sequence than at least one other artificial intelligence processing die.

[0007] In some implementations, each AI processing die of multiple identical AI processing dies is configured to receive data and perform AI calculations using the received data.

[0008] In some implementations, each of the multiple identical artificial intelligence processing dies is composed of a systolic array, and AI calculations are performed using the systolic array.

[0009] In some implementations, each artificial intelligence processing die of multiple identical artificial intelligence processing dies includes at least one host interface input block distinct from the inter-die input block and at least one host interface output block distinct from the inter-die output block.

[0010] In some implementations, each artificial intelligence processing die of multiple identical artificial intelligence processing dies includes at least one multiplication-accumulation unit (MAC unit).

[0011] In some implementations, each artificial intelligence processing die of multiple identical artificial intelligence processing dies includes at least one memory.

[0012] At least one aspect is directed toward a method for processing a neural network model. The method comprises receiving a first set of data related to a network in a first artificial intelligence processing die of an artificial intelligence processing unit. The first artificial intelligence processing die is associated with layers of the neural network. The method comprises performing an AI computation on the first set of data related to layers of the neural network associated with the first artificial intelligence processing die, using the first set of data related to the neural network. The method comprises transmitting the result data of the AI ​​computation on the first set performed in the first artificial intelligence processing die to a second artificial intelligence processing die of the artificial intelligence processing unit. The second artificial intelligence processing die is associated with different layers of the neural network from the first artificial intelligence processing die.

[0013] In some implementations, the first artificial intelligence processing die is associated with the input layer of a neural network.

[0014] In some implementations, the method comprises performing AI calculations on a second artificial intelligence processing die, using the result data of calculations performed on a first artificial intelligence processing die, related to layers of a neural network associated with the second artificial intelligence processing die. The method also comprises transmitting the result data of the AI ​​calculations performed on the second artificial intelligence processing die to the first artificial intelligence processing die as feedback.

[0015] In some implementations, the first artificial intelligence processing die and the second artificial intelligence processing die are arranged sequentially, with the first artificial intelligence processing die positioned before the second artificial intelligence processing die in terms of sequence.

[0016] In some implementations, the method comprises performing AI computations on a second set of data related to layers of a neural network associated with the first artificial intelligence processing die, using result data received as feedback from a second artificial intelligence processing die and data from a first set related to a neural network, on a first artificial intelligence processing die. The method also comprises transmitting the result data of the AI ​​computations on the second set to the second artificial intelligence processing die.

[0017] In some implementations, the second artificial intelligence processing die is associated with the output layer of the neural network.

[0018] In some implementations, the method involves the first artificial intelligence processing on a second artificial intelligence processing die. The method includes performing AI calculations related to the output layer of a neural network using the result data of calculations performed on a processing die. The method includes transmitting the result data of AI calculations performed on a second artificial intelligence processing die to a coprocessing unit that is communicatively connected to the artificial intelligence processing unit.

[0019] In some implementations, the first artificial intelligence processing die and the second artificial intelligence processing die include at least one multiplication-accumulation unit (MAC unit).

[0020] In some implementations, the first artificial intelligence processing die and the second artificial intelligence processing die include memory.

[0021] These and other aspects and implementations are described in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations, and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and implementations. The drawings provide illustrations and further understanding of the various aspects and implementations, and are incorporated herein and constitute a part of this specification.

[0022] The accompanying drawings are not intended to be drawn to scale. Like reference numerals and signs in the various drawings indicate like elements. For ease of understanding, not all components are described in every drawing.

Brief Description of the Drawings

[0023] [Figure 1A] FIG. is a diagram showing a system for processing neural network related tasks according to an exemplary implementation. [Figure 1B] FIG. is a diagram showing the functional logic of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 1C] FIG. is a diagram showing an example layout of a systolic array of an artificial intelligence processing die according to an exemplary implementation. [Figure 2A] FIG. is a diagram showing an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 2B] FIG. is a diagram showing an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 2C] FIG. is a diagram showing an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 2D] FIG. is a diagram showing an example layout of an artificial intelligence processing die of an artificial intelligence processing unit according to an exemplary implementation. [Figure 3]A flowchart showing an example of a method for constructing an artificial intelligence processing die according to an exemplary implementation. [Figure 4] A flowchart showing an example of a method for processing a neural network task based on a neural network model according to an exemplary implementation. [Figure 5] A block diagram showing a general architecture for a computer system that can be used to implement the components and methods of the systems described and illustrated herein according to an exemplary implementation.

Embodiments for Carrying Out the Invention

[0024] Detailed Description The present disclosure generally relates to apparatuses, systems, and mechanisms for processing the workload of neural networks. Efficient processing of neural networks utilizes application-specific integrated circuits (ASICs) designed for specific orders. However, the design of custom ASICs has several challenges, such as long design times and high non-recurring engineering costs, and these challenges are not limited to these, but they worsen when ASICs are produced in small quantities.

[0025] The challenges of using custom-designed ASICs can be overcome by designing standard dies configured to handle neural network tasks and interconnecting multiple such identical dies in a single ASIC chip package. The number of dies interconnected in a single chip package depends on the complexity or number of layers of the neural network being processed by the host computing device. In a package with multiple identical dies, different dies are associated with different layers of the neural network, thus increasing the efficiency of processing neural network-related tasks. By increasing or decreasing the number of dies in a single package based on the expected frequency of performing neural network tasks, it becomes possible to use standard dies across multiple products, allowing for more efficient amortization of the costs of long design times and significant extra-recurring engineering expenses.

[0026] Figure 1A shows a system 100 for performing a neural network computation task, relating to an exemplary implementation. System 100 comprises a main processing unit 101 and an artificial intelligence processing unit (AIPU) 102. System 100 is housed within a host computing device (not shown). Examples of host computing devices include, but are not limited to, servers and Internet of Things (IoT) devices. The AIPU 102 is a coprocessing unit of the main processing unit 101. The main processing unit 101 is communicably connected to the AIPU 102 via one or more communication channels, such as communication channels 104a, 104b, which are part of a communication system such as a bus. The main processing unit 101 includes a controller 105 and a memory 107. The memory 107 stores configuration data related to the subprocessing units of the main processing unit 101 and the coprocessing units connected to the main processing unit 101. For example, memory 107 may store configuration data related to AIPU 102. The main processing unit controller 105 is communicatively connected to memory 107 and is configured to select configuration data from memory 107 and transmit the configuration data to a coprocessing unit connected to the main processing unit 101 or to a subprocessing unit of the main processing unit 101. The selection and transmission of configuration data by the main processing unit controller 105 will be described in more detail below with reference to Figure 3.

[0027] AIPU102 is configured to process neural network computation tasks. AIPU102 includes a plurality of artificial intelligence processing dies (AIPDs) 103a, 103b, 103c, 103d, 103e, and 103f, collectively referred to herein as AIPD103. The AIPDs 103 are identical to one another. As described herein, an AIPD 103 is "identical" to another AIPD 103 if each AIPD 103 is manufactured using the same die design and the implementation of the hardware units on each AIPD 103 is identical to that of another AIPD 103. Thus, in this disclosure, two AIPDs 103 can be configured to process different layers of a neural network, provided that the die designs and hardware unit implementations of the two AIPDs 103 are identical, and are still considered identical. The number of AIPDs 103 included in AIPU102 may vary based on the number of layers of the neural network model to be processed by the host computing device. For example, if the host computing device is an Internet of Things (IoT) device such as a smart thermostat, the number of layers in the neural network model processed by the AIPU102 in the smart thermostat is likely to be fewer than the number of layers in the neural network model processed by the AIPU102 in a data center host computing device such as a data center server.

[0028] In a host computing device processing a simple neural network model, a single AIPD103 can efficiently handle neural network-related tasks on the host computing device. In a host computing device processing a more complex neural network model or a neural network model with multiple layers, multiple identical AIPD103s may be useful for efficiently handling neural network-related tasks. Therefore, in some implementations, the AIPU102 contains a single AIPD103, while in other implementations, the AIPU102 contains multiple identical AIPD103s.

[0029] As shown in Figure 1A, in an implementation where the AIPU102 includes multiple identical AIPD103s, each identical AIPD103 is linked to another identical AIPD103. Furthermore, each AIPD103 is associated with at least one layer of the neural network being processed by the AIPU102. The arrangement of the AIPD103s and multiple identical AIPD103s within the AIPU102 will be described in more detail below with reference to Figures 1B, 2A, and 2B.

[0030] Next, referring to Figure 1B, the functional logic of an implementation example of AIPD103 is shown. For the purpose of providing a clearer example, only the functional logic of AIPD103a is shown in Figure 1B, but since each of the AIPD103s is identical to one another, a person skilled in the art will understand that the functional logic of AIPD103b, 103c, 103d, 103e, and 103f is identical to the functional logic of AIPD103a. AIPD103a includes a host interface unit 113, a buffer 115, a controller 117, a buffer 119, a calculation unit 121, die-to-die input blocks 109a and 109b, and die-to-die output blocks 111a and 111b.

[0031] The host interface unit 113 includes at least one input / output (I / O) block (not shown). The I / O block includes a number of I / O pins (not shown). The I / O pins of the I / O block of the host interface unit 113 are configured bidirectionally so that the I / O block can receive data from a transmitting unit and send data to a receiving unit. Examples of transmitting and receiving units include, but are not limited to, a memory unit, a coprocessor of the main processing unit 101, or other integrated circuit components configured to send and receive data. The host interface unit 113 is configured to receive data from the main processing unit controller 105 via its I / O pins and to send data to the main processing unit controller 105, to the main processing unit 101, to itself, or directly to the memory 103 via its I / O pins. The host interface unit 113 stores the data received from the main processing unit controller 105 in a buffer 115.

[0032] Buffer 115 includes memory such as registers, dynamic random access memory (DRAM), static random access memory (SRAM), or other types of integrated circuit memory for storing data. The AIPD controller 117 is configured to retrieve data from buffer 115 and store data in buffer 115. The AIPD controller 117 is configured to operate in part on data transmitted from the main processing unit controller 105. If the data transmitted from the main processing unit controller 105 is configuration data, based on the configuration data, the AIPD controller 117 uses an inter-die input block used for communication between AIPD 103a and other AIPD 103s. and is configured to select the inter-die output block. Communication between AIPDs 103 will be described in more detail below with reference to Figures 2A, 2B, and 2C. When data transmitted from the main processing unit controller 105 is an instruction to perform a neural network task, the AIPD controller 117 is configured to store the data related to the neural network in a buffer 119 and to perform the neural network task using the input data stored in the buffer unit 119 and the compute unit 121. The buffer 119 includes memory such as registers, DRAM, SRAM, or other types of integrated circuit memory for storing data. The compute unit 121 includes multiple multiplication and accumulation units (MACs, not shown), multiple arithmetic logic units (ALUs) (not shown), and multiple shift registers (not shown), etc. Some of the registers in the buffer 119 are coupled to multiple ALUs in the compute unit 121 to establish a systolic array. The systolic array allows input values ​​to be read once and used for multiple different operations without storing the results before being used as input in a later operation. An example of such a systolic array configuration is shown in Figure 1C.

[0033] In Figure 1C, register 130 is contained in buffer 119, and the data from register 130 is the input for the first operation in ALU 132a. The result from ALU 132a is the input to ALU 132b, the result from ALU 132b is the input to ALU 132c, the result from ALU 132c is the input to ALU 132d, and so on. Such an arrangement and configuration distinguishes the AIPD 103 from a general-purpose computer that typically stores result data from one ALU in a memory unit before using that result data again. Furthermore, the arrangement shown in Figure 1C optimizes the AIPD 103 for computations related to the execution of artificial intelligence tasks (referred to herein as "AI computations") such as convolution, matrix multiplication, pooling, and element-wise vector operations. Moreover, by implementing the arrangement shown in Figure 1C, the AIPD 103 further optimizes power consumption and size in the execution of AI computations, thereby reducing the cost of the AIPU 102.

[0034] Referring again to Figure 1B, the computing unit 121 performs AI calculations using the input data and weights selected for the neural network and transmitted from a weight memory unit (not shown). In some implementations, the computing unit 121 includes an activation unit 123. The activation unit 123 may include multiple ALUs and multiple shift registers and may be configured to apply an activation function and a nonlinear function to the results of the AI ​​calculations. The activation function and nonlinear function applied by the activation unit 123 can be implemented in hardware, firmware, software, or a combination thereof. The computing unit 121 transmits the data resulting after applying the activation function and / or nonlinear function to the buffer 119 to the buffer 119 that stores the data. The AIPD controller 117 uses an inter-die output block configured for inter-die communication to transmit the output data from the computing unit 121 stored in the buffer 119 to the AIPD 103, which is communicatively connected to the AIPD 103a. The configuration data received from the main processing unit controller 105 determines the inter-die communication path between the two AIPDs 103. For example, if the configuration data received by AIPD 103a indicates that the inter-die output block 111a (as shown in Figure 1B) should be used for inter-die communication, the AIPD controller 117 uses the inter-die output block 111a to send data to the other AIPD 103. Similarly, if the configuration data indicates that the input block 109b (as shown in Figure 1B) should be used for inter-die communication, the AIPD controller 117 selects the input block 109b as the inter-die input block for receiving data from the other AIPD 103, and reads and processes the data received by the input block 109b.

[0035] Each die-to-die input and die-to-die output block of the AIPD103 contains multiple pins. The pins of the die-to-die output block of an AIPD103 can be connected by electrical interconnections to the corresponding pins of the die-to-die input blocks of other AIPD103s. For example, as shown in Figure 2A, the pins of output block 111a of AIPD103a are connected by electrical interconnections to the input block of AIPD103b. The electrical interconnections between the pins of die-to-die output and die-to-die input blocks of different AIPD103s are of equal length.

[0036] The connection between the die-to-die output block of one AIPD103 and the die-to-die input block of another AIPD103 is established by an electrical interconnection, while the selection of a particular die-to-die output block of the AIPD103 and the transmission of specific signals or data to specific pins of the die-to-die output block are programmable or modifiable based on configuration data received by the AIPD103 from the main processing controller 105. By selecting different output blocks of the AIPD103, the AIPU102 can be configured to fulfill different requirements for different neural networks, including feedback loops between different layers of the neural network, although these requirements are not limited to these. Thus, various sets of neural networks can be implemented using the same AIPU102, resulting in reduced design time costs and efficient amortization of non-regular engineering costs. The configurations of the AIPD103 and AIPU102 are described in further detail below with reference to Figures 2A, 2B, and 3.

[0037] As described above, each of the multiple AIPD103s is associated with at least one layer of a neural network configured to be processed by the AIPU102. The main processing unit 101 contains configuration data that constitutes the AIPD103s and the AIPU, such as the AIPU102. The configuration data is associated with the neural network model selected to be processed by the AIPU. The configuration data identifies the association between the AIPD103s and the layers of the neural network being processed by the AIPU. Based on the configuration data associated with the neural network being processed by the AIPU, the main processing unit controller 105 associates the AIPD103s with the layers of the neural network. In some implementations, the main processing unit controller 105 stores the association between the AIPD103s and the layers of the neural network in a storage device such as memory 107 (shown in Figure 1A). The main processing unit controller 105 transmits the configuration data associated with the AIPD103s to the corresponding AIPD103s. The association of AIPD103 with the layers of a neural network is partially based on the requirements of the neural network model being processed by AIPU102. For example, if the neural network includes a feedback loop between two layers of the neural network, the AIPD103 associated with these two layers can be selected partly based on whether the die-to-die output block of the first AIPD103 and the die-to-die input block of the second AIPD103 are electrically interconnected. An example of such an arrangement of multiple AIPD103s is illustrated with reference to Figure 2A.

[0038] Figure 2A shows an example of the arrangement of multiple AIPD103 within an AIPU such as AIPU102. In Figure 2A, AIPU102 contains six AIPD103 (AIPD103a, 103b, 103c, 103d, 103e, 103f) and processes a neural network having six layers, including a feedback loop between the last layer and the first layer of the neural network. AIPD103a includes die-to-die input blocks 109a, 109b, die-to-die output blocks 111a, 111b, and a host interface unit 113. AIPD103b includes die-to-die input blocks 221a, 221b, and die-to-die output AIPD103c includes blocks 223a, 223b, and a host interface unit 214. AIPD103c includes die-to-die input blocks 225a, 225b, die-to-die output blocks 227a, 227b, and a host interface unit 215. AIPD103d includes die-to-die input blocks 229a, 229b, die-to-die output blocks 231a, 231b, and a host interface unit 216. AIPD103e includes die-to-die input blocks 233a, 233b, die-to-die output blocks 235a, 235b, and a host interface unit 217. AIPD103f includes die-to-die input blocks 237a, 237b, die-to-die output blocks 239a, 239b, and a host interface unit 218.

[0039] Each AIPD103 is associated with a specific layer of the neural network, and as described above, the association of an AIPD103 with a layer of the neural network is partially based on the features associated with that layer of the neural network. Since the neural network in Figure 2A requires a feedback loop between the last layer and the first layer of the neural network, the last layer and the first layer of the neural network should be associated with an AIPD103, and in the AIPD103, the die output block of the AIPD103 associated with the last layer of the neural network is electrically interconnected with the die input block of the AIPD103 associated with the first layer of the neural network. As shown in Figure 2A, the die output block 231a of AIPD103d is electrically interconnected with the die input block 109b of AIPD103a, so such an arrangement can be made by associating AIPD103a with the first layer and AIPD103d with the sixth layer. Therefore, AIPD103b, 103c, 103f, and 103e are associated with the second, third, fourth, and fifth layers of the neural network, respectively. In Figure 2A, the order of AIPD103 is such that AIPD103a is in the first position, AIPD103b is in the second position, AIPD103c is in the third position, AIPD103f is in the fourth position, AIPD103e is in the fifth position, AIPD103d is in the sixth position, and AIPD103a is in the seventh position. The order of communication of neural network-related data between AIPD103 is as shown in 201a, 201b, 201c, 201d, 201e, 201f, starting with 103a, then to 103b, then to 103c, 103f, 103e, 103d, and then back to 103a, incorporating a feedback layer between the sixth and first layers of the neural network. As described herein, “neural network-related data” includes, but is not limited to, computation result data such as the output of the computation unit 121, parameter weight data, and other neural network parameter-related data.

[0040] The AIPD controller of the AIPD 103 associated with the output layer of the neural network is configured to transmit result data from the output layer to the main processing unit 101. For example, if the AIPD associated with the output layer is 103d, the AIPD controller 216 is configured to transmit result data from AIPD 103d to the main processing unit 101. In some implementations, a single AIPD 103 is configured to receive the initial input data of the neural network from the main processing unit 101 and transmit result data from the last layer of the neural network to the main processing unit 101. For example, in Figure 2A, if AIPD 103a receives the initial input data of the neural network from the main processing unit 101 and result data from AIPD 103d, the AIPD associated with the last layer of the neural network, the AIPD controller 113 of AIPD 103a can be configured to transmit result data from AIPD 103d, received at the inter-die input block 111b, to the main processing unit 101.

[0041] Using the same AIPD103 described above, it is possible to process neural networks different from the neural network described with reference to Figure 2A. For example, if the neural network has a feedback loop between the sixth and third layers of the neural network, the sixth and third layers should be associated with the AIPD103, where the die output block of the AIPD103 associated with the sixth layer of the neural network is electrically interconnected with the die input block of the AIPD103 associated with the third layer of the neural network. Furthermore, each of the AIPD103s associated with different layers of the neural network is electrically interconnected with at least one die input block of other AIPD103s associated with later layers of the neural network. For example, an AIPD103 associated with the first layer should have an inter-die output block electrically interconnected with the inter-die input block of an AIPD103 associated with the second layer of the neural network; an AIPD103 associated with the second layer should have an inter-die output block electrically interconnected with the inter-die input block of an AIPD103 associated with the third layer of the neural network; an AIPD103 associated with the third layer should have an inter-die output block electrically interconnected with the inter-die input block of an AIPD103 associated with the fourth layer of the neural network; an AIPD103 associated with the fourth layer should have an inter-die output block electrically interconnected with the inter-die input block of an AIPD103 associated with the fifth layer of the neural network; and an AIPD103 associated with the fifth layer should have an inter-die output block electrically interconnected with the inter-die input block of an AIPD103 associated with the sixth layer of the neural network. Such neural network processing can be performed using the AIPD103 configuration shown in Figure 2B.

[0042] Figure 2B shows different arrangement examples of AIPD103 within the AIPU. In Figure 2B, AIPU250 includes AIPD103a, 103b, 103c, 103d, 103e, and 103f. Within the AIPU250, the die output block 111a of AIPD103a is electrically interconnected with the die input block 221a of AIPD103b, the die output block 111b of AIPD103a is electrically interconnected with the die input block 229a of AIPD103d, the die output block 223b of AIPD103b is electrically interconnected with the die input block 109b of AIPD103a, the die output block 223a of AIPD103b is electrically interconnected with the die input block 225a of AIPD103c, and the die output block 227b of AIPD103c is electrically interconnected with the die input block 237a of AIPD103f. The die output block 239a of AIPD103f is electrically connected to the die input block 225b of AIPD103c, the die output block 239b of AIPD103f is electrically interconnected to the die input block 233b of AIPD103e, the die output block 235a of AIPD103e is electrically interconnected to the die input block 221b of AIPD103b, the die output block 235b of AIPD103e is electrically interconnected to the die input block 229b of AIPD103d, and the die output block 231a of AIPD103d is electrically interconnected to the die input block 233a of AIPD103e.

[0043] In Figure 2B, AIPD103f is associated with the sixth layer of the neural network, and AIPD103e is associated with the third layer of the neural network. AIPD103a, 103d, 103b, and 103c are associated with the first, second, fourth, and fifth layers of the neural network, respectively. The AIPD controller 113 transmits the computation result data from AIPD103a to AIPD103d, an AIPD103 associated with the second layer of the neural network, and to the inter-die output block of AIPD103a, which is electrically interconnected to the inter-die input block 229a of AIPD103d. It is configured to transmit using lock 111b. The AIPD controller 216 of AIPD103d is configured to transmit result data from AIPD103d to AIPD103e, an AIPD103 associated with the third layer of the neural network, using the die output block 231a electrically interconnected to the die input block 233a of AIPD103e. The AIPD controller 217 of AIPD103e is configured to transmit result data from AIPD103e to AIPD103b, an AIPD103 associated with the fourth layer of the neural network, using the die output block 235a of AIPD103e electrically interconnected to the die input block 221b of AIPD103b. The AIPD controller 214 of AIPD103b is configured to transmit result data from AIPD103b to AIPD103c, an AIPD103 associated with the fifth layer of the neural network, using an inter-die output block 223a electrically interconnected to the inter-die input block 225a of AIPD103c. The AIPD controller 215 of AIPD103c is configured to transmit result data from AIPD103c to AIPD103f, an AIPD103 associated with the sixth layer of the neural network, using an inter-die output block 227b of AIPD103c electrically interconnected to the inter-die input block 237a of AIPD103f. The AIPD controller 218 is configured to transmit feedback data from AIPD 103f to AIPD 103e, an AIPD 103 associated with the third layer of the neural network, using the inter-die output block 239b of AIPD 103f, which is electrically interconnected to the inter-die input block 233b of AIPD 103e. The AIPD controller 218 of AIPD 103f is further configured to transmit result data from AIPD 103f to the main processing unit 101 if AIPD 103f is associated with the output layer of the neural network.In Figure 2B, the AIPD103s are arranged in the following order: AIPD103a is in the first position, AIPD103d is in the second position, AIPD103e is in the third position, AIPD103b is in the fourth position, AIPD103c is in the fifth position, AIPD103f is in the sixth position, and AIPD103e is in the seventh position. The order of communication of neural network-related data between the AIPD103s in Figure 2B is as shown by 202a, 202b, 202c, 202d, 202e, and 202f, starting with 103a, then 103d, 103e, 103b, 103c, 103f, and then sending feedback data to 103e.

[0044] Therefore, the same AIPD can be used to process different neural networks with different neural network requirements. Thus, the design of a single artificial intelligence processing die (AIPD) can be used for processing and running different neural networks with different requirements, thereby reducing design time-related costs and efficiently amortizing non-recurring engineering costs.

[0045] Furthermore, by modifying the configuration data associated with the AIPU and / or the configuration data associated with the AIPD of the AIPU, a single AIPU can be used to process different neural networks. For example, in Figure 2B, if a neural network with four layers is processed by the AIPU250, the configuration data associated with the AIPU250 and / or the configuration data associated with the AIPD103 of the AIPU250 can be modified to associate AIPD103a with the first layer of the neural network, AIPD103b with the second layer of the neural network, AIPD103c with the third layer of the neural network, and AIPD103f with the fourth layer of the neural network. The electrical interconnection between the die output block and die input block of these AIPD103s has been described above. When the AIPU250 and the AIPD103 of the AIPU250 are reconfigured, the main processing unit The controller 105 transmits input data related to the neural network to AIPD 103a, associated with the first layer of the neural network. Based on the modified configuration data and input data to the neural network associated with AIPD 103a, AIPD 103a performs calculations related to the new first layer of the neural network, including AI calculations, and transmits the result data to AIPD 103b using the die output block 111a. As described herein, “calculations related to a layer of the neural network” includes AI calculations related to that layer of the neural network. Based on the result data received from AIPD 103 at the die input block 221a and the modified configuration data associated with AIPD 103b, AIPD 103b performs calculations related to the second layer of the neural network, including AI calculations. AIPD 103b transmits the result data to AIPD 103c using the die output block 223a. AIPD103c performs calculations related to the third layer of the neural network, including AI calculations, based on the result data received from AIPD103b at the inter-die input block 225a and the modified configuration data associated with AIPD103c, and transmits the result data to AIPD103f using the inter-die output block 227b. AIPD103f performs calculations related to the fourth layer of the neural network, including AI calculations, based on the result data received from AIPD103c at the inter-die input block 237a and the modified configuration data associated with AIPD103f. AIPD103, associated with the last layer of the neural network, is configured to transmit the result data from AIPD103f to the main processing unit 101. Therefore, a single AIPU can be reprogrammed to process different neural networks by changing the configuration data associated with the AIPU and / or the configuration data of the AIPD of the AIPU.Therefore, the significant extra-regular engineering costs associated with the use of custom ASICs can be more efficiently amortized, and the design time costs associated with designing custom ASICs to handle this particular neural network task can be further reduced.

[0046] In some implementations, at least one die-input block and at least one die-output block are located on one edge of the AIPD103, and at least one die-output block and at least one die-input block are located on the other edge of the AIPD103. For example, as shown in Figure 2A, one die-input block and one die-output block are located on the top edge of the AIPD103, and the other die-output block and die-input block are located on the bottom edge of the AIPD103. In some implementations, all die-input blocks are located on one edge of the AIPD103, and all die-output blocks are located on the other edge of the AIPD103b. An example of such an arrangement of die-input and die-output blocks is shown in Figure 2C.

[0047] In Figure 2C, all die-to-die input blocks are located on the top edge of the AIPD 103, and all die-to-die output blocks are located on the bottom edge of the AIPD 103. In some implementations, the orientation of some AIPD 103s is offset by a certain distance or angle relative to the orientation of other AIPD 103s in order to achieve equal-length electrical interconnections between the AIPD 103s and to achieve a more efficient size for the AIPU including the AIPD 103s shown in Figure 2C. For example, as shown in Figure 2C, AIPD 103b and 103e are rotated 180 degrees relative to the orientations of AIPD 103a, 103d, 103c, and 103f. By rotating AIPD103 by 180 degrees, the die-to-die input and output blocks of AIPD103b and 103e are positioned adjacent to the die-to-die output and input blocks of AIPD103a, 103d, 103c, and 103f, thereby ensuring that the length of the electrical interconnections between all AIPD103s is equal, and between AIPD103b or 103e and any adjacent AIPD103s. No additional space is required for the electrical interconnection between the input block and the die output block.

[0048] In Figure 2C, an AIPU having the AIPD103 configuration shown in Figure 2C can process a neural network similar to the AIPU described above. For example, a neural network with six layers and no interlayer feedback loops can be processed by the AIPD103 configuration shown in Figure 2C, by associating AIPD103a with the first layer of the neural network, AIPD103d with the second layer, AIPD103e with the third layer, AIPD103b with the fourth layer, AIPD103c with the fifth layer, and AIPD103f with the sixth layer. In the arrangement of AIPD103 in Figure 2C, AIPD103a is in the first position, AIPD103d is in the second position, AIPD103e is in the third position, AIPD103b is in the fourth position, AIPD103c is in the fifth position, and AIPD103f is in the sixth position. Communication between AIPD103s begins with AIPD103a, followed by AIPD103d, 103e, 103b, 103c, and 103f in that order.

[0049] One of the advantages of the AIPD design and implementation described herein is that any number of AIPDs can be contained within a single AIPU package. The number of AIPDs within a single AIPU package is limited only by the size of the AIPU package, and not by the size of the AIPD dies. Therefore, it is possible to include an N×N arrangement of AIPDs in a single AIPU package, as shown by the arrangement of AIPD11 to AIPDNN in Figure 2D. AIPD11 to AIPDNN in Figure 2D are designed and configured similarly to AIPD103 described above.

[0050] The main processing unit controller 105 is configured to send the initial input data of the neural network to the AIP 103 associated with the first layer (input layer) via the host interface unit of the AIPD 103. For example, as shown in Figures 2, 2B, and 2C, the AIPD 103 associated with the first layer of the neural network is AIPD 103a, and the main processing unit controller 105 sends the initial input data to AIPD 103a via the host interface unit 113. In some implementations, the last AIPD 103 in the communication sequence is configured to send the result data back to the main processing unit 101 using the AIPD's host interface unit. In some implementations, the AIPD 103 associated with the last layer of the neural network is configured to send the result data back to the main processing unit 101. For example, in Figure 2A, as described above, the last AIPD103 in the communication sequence is AIPD103a, so in some implementations, the AIPD controller 117 of AIPD103a is configured to transmit result data to the main processing unit 101 using the host interface unit 113. Similarly, in Figure 2B, AIPD103f is an AIPD103 associated with the last layer of the neural network, and in some implementations, the AIPD controller 218 of AIPD103f is configured to transmit result data to the main processing unit 101 using the host interface unit of AIPD103f. An example of how to configure AIPD103 for neural network processing will be explained with reference to Figure 3.

[0051] Figure 3 is a flowchart of an example 300 of a method for configuring an AIPU to process a neural network model. In the main processor, method 300 comprises receiving input and configuring the AIPU (stage 302). Method 300 comprises selecting AIPU configuration data (stage 304). Method 300 comprises sending the configuration data to the AIPD103 of the AIPU (stage 304). Method 300 includes (stage 306). In each AIPD 103, Method 300 includes receiving configuration data (stage 308). Method 300 includes configuring the AIPD 103 based on the configuration data (stage 310). Method 300 includes sending a response to the main processing unit 101 (stage 312).

[0052] Method 300 comprises receiving inputs and configuring the AIPU in the main processing unit 101 (stage 302). In response to receiving inputs and configuring the AIPU, Method 300 comprises selecting AIPU configuration data for each AIPD 103 in the AIPU (stage 304). The main processing unit controller 105 of the main processor 101 selects the configuration data associated with the AIPU. In selecting the configuration data associated with the AIPU, the main processing unit controller 105 selects the configuration data associated with each of the AIPD 103 in the AIPU. Different configuration data may specify different values ​​to configure the AIPD 103 for neural network processing. The AIPD 103 includes, but is not limited to, the die-to-die output block and die-to-die input block of the associated AIPD 103, which are configured for sending and receiving neural network-related data between the associated AIPD 103 and other AIPD 103s, as well as for output data to the pins of the die-to-die output block 103, and for mapping neural network-related data such as parameters, parameter weight data, and the number of parameters. The values ​​specified by the configuration data are based on the layer of the neural network to which the corresponding AIPD103 is associated. Therefore, the configuration data values ​​associated with one AIPD103 may differ from the configuration data values ​​associated with a different AIPD103.For example, if the first layer of a neural network being processed by the AIPU requires a first set of weight values ​​to be used for the computational task of the first layer of the neural network, and the second layer of the neural network requires a second set of weight values, different from the first set of weight values, to be applied during the computational task of the second layer, then the configuration data associated with the AIPD103 associated with the first layer of the neural network specifies the weight values ​​corresponding to the first set of weight values, while the configuration data associated with the AIPD103 associated with the second layer of the neural network specifies the weight values ​​corresponding to the second set of weight values.

[0053] The die output block of AIPD103 specified in the configuration data for transmitting neural network-related data to the AIPD103 associated with the next layer of the neural network is partially based on the position of AIPD103 relative to the AIPD103 associated with the next layer of the neural network. For example, if AIPD103a is associated with the first layer of the neural network and AIPD103b is associated with the next layer of the neural network, then the die output block of AIPD103a specified in the configuration data for AIPD103a is the die output block electrically interconnected to the die input block of AIPD103b, which is die output block 111a, as shown in Figures 2A, 2B, and 2C. Similarly, if AIPD103d is associated with the next layer after the layer associated with AIPD103a, the inter-die output block selected to transmit neural network-related data and specified in the configuration data of AIPD103a is the inter-die output block electrically interconnected to the inter-die input block of AIPD103d, which is the inter-die output block 111b, as shown in Figures 2A, 2B, and 2C.

[0054] Each AIPD103 is associated with a unique identifier, and in some implementations, the configuration data of the AIPD103 is associated with that unique identifier, and the main processing unit controller 105 is configured to select the configuration data of the AIPD103 based on the unique identifier associated with the AIPD103.

[0055] Method 300 includes sending selected configuration data to the AIPD 103 (stage 306). As described above, the main processing unit controller 105 sends the configuration data to the AIPD 103 via the host interface unit of the AIPD 103, such as the host interface unit 113 of the AIPD 103a. In some implementations, the main processing unit controller 105 is configured to periodically check whether the configuration data for any AIPD 103 has been updated, and in response to the update of the configuration data for the AIPD 103, the main processing unit controller 105 sends the updated configuration data to the specific AIPD 103. In some implementations, the main processing unit controller 105 sends instructions to the AIPD 103 to configure the AIPD 103 based on the received configuration data. In some implementations, the configuration data is stored in the host computing device memory, and the AIPD 103 is configured to read the data stored in the host computing device memory. In some implementations, the main processing unit controller 105 sends commands to the AIPD 103 to read configuration data from the host computing device memory and configure the AIPD 103 based on the configuration data.

[0056] Method 300 comprises, in each AIPD 103, receiving configuration data (stage 308) and configuring the AIPD 103 based on the received configuration data (stage 310). As described above, the AIPD controller of an AIPD 103, such as the AIPD controller 117 of AIPD 103a, is configured to select an inter-die input block and an inter-die output block and to configure these blocks to send and receive data to and from other AIPD 103s based on the received configuration data. The AIPD controller of an AIPD 103 is also configured to send specific output data of the AIPD 103, such as the output from the computing unit 121, to a specific pin of the selected inter-die output block that has been selected to send neural network-related data to other AIPD 103s, based on the received configuration data. Furthermore, the AIPD controller of an AIPD 103 is configured to store neural network-related data, such as parameter weight data, in a storage device such as a buffer 119 and to use the neural network-related data between layers of the neural network associated with the AIPD 103 and related computations.

[0057] Method 300 includes sending a response signal to the main processor 101 in each AIPD 103 (stage 312). The AIPD 103 sends a response signal to the main processor 101 using a host interface unit such as the host interface unit 113 of the AIPD 103a. The response sent to the main processor 101 informs the main processor that the configuration of the AIPD 103 is normal. In some implementations, if an error occurs during the configuration of the AIPD 103, the AIPD 103 sends an error message to the main processor 101 using the host interface unit. Once the required AIPD 103 is successfully configured, the AIPU is ready to process neural network-related tasks. The main processing unit controller 105 puts the neural network task into the AIPU to execute the neural network task. It is sent to [the appropriate destination]. An example of how AIPU processes neural network tasks is described below, with reference to Figure 4.

[0058] Figure 4 is a flowchart of an example 400 of a method for processing neural network-related tasks using an AIPU. In the main processor 101, method 400 includes identifying a neural network task (stage 402). Method 400 includes sending initial data or input data related to the neural network to the AIPU (stage 404). In the AIPU, method 400 includes receiving initial data related to the neural network in a first AIPD 103 associated with the input layer of the neural network (stage 406). Method 400 includes performing calculations related to the layers of the neural network associated with the first AIPD 103 in the first AIPD 103 using the initial data and any neural network-related data received along with the configuration data of the first AIPD 103 (stage 408). Method 400 includes sending the results of the calculations to a second AIPD (stage 410). Method 400 includes performing calculations related to the layers of the neural network associated with the second AIPD 103 using result data received from the first AIPD (stage 412). In some implementations, Method 400 includes sending the results of the calculations in the second AIPD 103 as feedback to the first AIPD 103 (stage 414). Method 400 includes sending the results of the neural network from the AIPU to the main processor (stage 416). Method 400 includes sending the results of the neural network to the user in the main processor (stage 418).

[0059] Method 400 includes identifying a neural network task in the main processor 101 (stage 402). The main processing unit controller 105 is configured to identify whether the requested task is a neural network-related task. In some implementations, the request message or data for the requested task carries specific indicators, such as high or low bits in a particular area of ​​the message, indicating that the requested task is a neural network-related task, and the main processing unit controller 105 is configured to determine whether the requested task is a neural network task based on these specific indicators.

[0060] Method 400 includes the main processor 101 transmitting input data for the neural network to the AIPU (stage 404). The main processing unit controller 105 of the main processing unit 101 retrieves the input data from the memory of the host computing device and transmits it to the AIPD 103 associated with the initial layer or input layer of the neural network being processed by the AIPU. The main processing unit controller 105 identifies the AIPD 103 associated with the input layer of the neural network based on the configuration data associated with each AIPD 103. In some implementations, the identifier of the AIPD 103 associated with the input layer of the neural network is stored in memory or a storage unit such as a register or buffer, and the main processing unit controller 105 determines the AIPD 103 associated with the input layer based on the identifier stored in memory or the storage unit. In an implementation where the AIPD103 is configured to read data stored in the host computing device's memory, the main processing unit controller 105 sends instructions to the AIPD103 associated with the input layer of the neural network to retrieve input data for the neural network from the host computing device's memory.

[0061] Method 400 comprises receiving input data related to a neural network in a first AIPD103 associated with the input layer of a neural network such as AIPD103a as described with reference to Figures 2A, 2B, and 2C (stage 406). Method 400 comprises performing calculations related to the layers of the neural network associated with the first AIPD103 in the first AIPD103 using the initial data received in the first AIPD103 and other neural network-related data received during the configuration of the first AIPD103 (stage 408). The controller of the first AIPD103 determines the calculations to be performed based on the associated neural network layers. For example, if the first layer of the neural network performs matrix multiplication by applying a weight matrix to the input data, during the configuration of the AIPD103, the weight matrix is ​​sent to the first AIPD103 and stored in the AIPD103's buffer. The AIPD controller of the first AIPD103 is configured to send the weight matrix to the computing unit of the first AIPD103 and perform matrix multiplication using the weight matrix and input data. In some implementations, the calculation to be performed is specified in the configuration data received by the first AIPD103, and based on the specified calculation, the controller of the first AIPD103 sends data to the appropriate computing unit of the AIPD103, such as the computing unit 121 of the AIPD103a.

[0062] Method 400 includes, in the first AIPD 103, transmitting the results of the calculation in the first AIPD 103 to the second AIPD 103 (stage 410). The second AIPD 103 is associated with a different neural network layer than the first AIPD. Method 400 also includes, in the second AIPD 103, performing calculations related to the neural network layer associated with the second AIPD 103 using the result data received from the first AIPD 103 and other neural network-related data (stage 412). In some implementations, the controller of the AIPD 103 performing the calculations can retrieve further data for the calculations, such as parameter weight data used in the AI ​​calculations, from the memory of the host computing device.

[0063] In an implementation where the neural network model being processed by the AIPU includes a feedback loop between two or more layers of the neural network, and the second AIPD103 and the first AIPD103 are associated with layers of the neural network that include a feedback loop between them, Method 400 includes sending the computation result data in the second AIPD103 as feedback to the first AIPD103 (Stage 414). If there is no feedback loop between the layers associated with the second AIPD103 and the first AIPD103, Method 400 includes sending the results of the neural network from the AIPU to the main processing unit 101 (Stage 416). The controller of the AIPD103 associated with the output layer of the neural network sends the results of the neural network to the main processor 101 using a host interface such as the host interface unit 113 of the AIPD103a. For example, in Figure 2A, AIPD103a is an AIPD103 associated with the output layer of the neural network in Figure 2A, and therefore, the AIPD controller 117 of AIPD103a transmits the result data to the main processor 101 using the host interface unit 113. Similarly, in Figure 2B, AIPD103f is associated with the output layer of the neural network in Figure 2B, and the AIPD controller of AIPD103f transmits the result data to the main processor 101 using the host interface unit of AIPD103f.

[0064] Method 400 involves the main processing unit 101 transmitting the neural network results received from the AIPU to the requesting side of the neural network task. (Stage 418) is included. As used herein, “the requesting party of a neural network task” may be another process within the host computing device or an end user of the host computing device. Although only two AIPD103s are shown in Figure 4 for clarity and to provide a clear example, the number of AIPD103s used in executing a neural network task is determined, at least in part, by the volume of neural network tasks expected to be performed by the host computing device.

[0065] Figure 5 is a block diagram showing an overall architecture for a computer system 500 that may be used to implement the elements and methods of the system described and illustrated herein, relating to an exemplary implementation. The computing system 500 can be used to implement the host computing device described above. The computing system 500 may also be used to implement the configuration of the AIPU method 300 shown in Figures 3 and 4 and to process neural network tasks using the AIPU method 400.

[0066] In a broader sense, the computing system 510 includes at least one processor 550 that operates according to instructions, and one or more memory devices 570 or 575 that store instructions and data. An illustrated example of the computing system 510 includes one or more processors 550 connected via a bus 515 to at least one network interface controller 520 having one or more network interface ports 522 connected to a network (not shown), an AIPU 590, memory 570, and other elements 580, such as an input / output (I / O) interface 530. Generally, the processor 550 executes instructions received from memory. The illustrated processor 550 incorporates or is directly connected to a cache memory 575.

[0067] More specifically, the processor 550 may be any logic circuit that processes instructions, for example, instructions retrieved from memory 570 or cache 575. In many embodiments, the processor 550 is a microprocessor unit or a special-purpose processor. The computing device 500 may be any processor or set of processors capable of operating as described herein. In some implementations, the processor 550 is capable of executing certain stages of method 300 shown in Figure 3, such as stages 302, 304, and 306, and certain stages of method 400 shown in Figure 4, such as stages 402, 404, and 418. The processor 550 may be a single-core or multi-core processor. The processor 550 may be multiple processors. In some implementations, the processor 550 can be configured to perform multi-threaded operation. In some implementations, the processor 550 may host one or more virtual machines or containers in conjunction with a hypervisor or container manager for managing the operation of these virtual machines or containers. In such implementation examples, method 300 shown in Figure 3 and method 400 shown in Figure 4 can be implemented within a virtual environment or container environment provided on the processor 550.

[0068] Memory 570 may be a device suitable for storing computer-readable data. Memory 570 may be a device having a fixed storage device or a device for reading a removable storage medium. Examples include all forms of non-volatile memory, media and memory devices, semiconductor memory devices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magnetic disks, magneto-optical disks, and optical disks (e.g., CD-ROM, DVD-ROM, and Blu-ray® disks). The computing system 500 may have any number of memory devices 570. In some implementations, memory 570 is as shown in Method 300 in Figure 3 and Figure 4. The instructions may include those corresponding to method 400 shown. In some implementations, memory 570 supports virtual memory or container memory accessible in a virtual machine or container execution environment provided by the computing system 510.

[0069] Cache memory 575 is generally a form-skinned computer memory located very close to the processor 550 for fast read times. In some implementations, cache memory 575 is part of the processor 550 or on the same chip as the processor 550. In some implementations, there are multiple levels of cache 575, such as L2 and L3 cache layers.

[0070] The network interface controller 520 manages data exchange via the network interface 522 (also called the network interface port). The network interface controller 520 handles the physical and data link layers of the OSI model for network communication. In some implementations, some of the network interface controller's tasks are handled by the processor 550. In some implementations, the network interface controller 520 is part of the processor 550. In some implementations, the computing system 510 has multiple network interface controllers 520. The network interface 522 is a connection point for a physical network link. In some implementations, the network interface controller 520 supports wireless network connectivity, and the interface port 522 is a wireless receiver / transmitter. Generally, the computing device 510 exchanges data with other computing devices via a physical or wireless link to the network interface 522. The network interface 522 may link directly to other devices, or it may be connected via an intermediate device that connects the computing device 510 to a network such as the Internet, such as a hub, bridge, switch, or router. In some implementations, the network interface controller 520 implements network protocols such as Ethernet®.

[0071] Other components 580 may include an I / O interface 530, an external serial device port, and further coprocessors. For example, the computing system 510 may include an interface (e.g., a Universal Serial Bus (USB) interface) for connecting input devices (e.g., a keyboard, microphone, mouse, or other pointing device), output devices (e.g., a video display, speaker, or printer), or additional memory devices (e.g., a portable flash drive or external media drive). In some implementations, other components 580 may include additional coprocessors, such as a numerical coprocessor, to assist the processor 550 with high-precision or complex calculations.

[0072] Implementations of the subject matter and operations of the inventions described herein may be implemented in digital electronic circuits, or in computer software, firmware, or hardware contained on a tangible medium, or in one or more combinations thereof, including structures disclosed herein and their structural equivalents. Implementations of the subject matter of the inventions described herein may be implemented as one or more computer programs contained on a tangible medium, that is, as one or more modules of computer program instructions encoded on one or more computer storage media for execution by a data processing device or for controlling the operation of a data processing device. The computer storage media may be a computer-readable storage device, a computer-readable storage board, a random or serial access memory array or element, or one or more combinations thereof. Computer storage media can be a combination of these, or included in them. Computer storage media can also be one or more separate components or media (for example, multiple CDs, disks, or other storage devices), or included in them. Computer storage media may also be tangible, non-temporary computer storage media.

[0073] The operations described herein can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other senders. The operations may be performed within the native environment of the data processing device or within one or more virtual machines or containers hosted by the data processing device.

[0074] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computer environment. Computer programs may, but do not, correspond to files in a file system. A program may be stored in a single file dedicated to it, as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in multiple collaborative files (e.g., a file that holds one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on a single computer, or on one or more virtual machines or containers located in one place or distributed across multiple locations and interconnected by a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0075] The processes and logic flows described herein may be carried out by one or more programmable processors executing one or more computer programs to perform actions by acting on input data and producing outputs. The processes and logic flows may also be carried out by dedicated logic circuits, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), and the devices may also be implemented as dedicated logic circuits.

[0076] This specification includes details of many specific examples, which should not be construed as limitations on the invention or claims, but rather as descriptions of features specific to a particular example of a particular invention. Some features described herein within the framework of separate examples may be realized in combination in a single example. Conversely, various features described within the framework of a single example may be realized in multiple separate examples or in any suitable partial combination. Also, features have been described above as operating in several combinations and may have been initially claimed in that manner, but one or more features from a claimed combination may, in some cases, be removed from the combination, and the claimed combination may cover a partial combination or a variation of a partial combination.

[0077] Similarly, while actions are shown in a specific order in the diagrams, this should not be understood as meaning that such actions must be performed in a specific or sequential order shown to achieve the desired result, or that all shown actions must be performed to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, The separation of various system components in the above implementation examples should not be understood as a requirement for all implementations. It should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0078] The use of "or" can be interpreted as encompassing any term used with "or" to refer to one, two or more, or all of the terms listed. Expressions such as "first," "second," and "third" are not necessarily intended to indicate order, but are generally used to distinguish similar or identical items or elements.

[0079] Various modifications to the examples described herein will be obvious to those skilled in the art, and the general principles described herein can be applied to other examples without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the examples described herein, but should be interpreted in the broadest sense to be consistent with the principles and novel features disclosed herein.

Claims

1. A system for processing a neural network computation task, An artificial intelligence processing die that corresponds to a neural network layer, The artificial intelligence processing die is connected to a processor, and the processor is Select an input representing the configuration used to implement the neural network layer in the artificial intelligence processing die, The selected input is configured to be transmitted to the artificial intelligence processing die, The artificial intelligence processing die is configured to perform neural network processing operations on the neural network layer according to the configuration represented by the selected input.

2. The system according to claim 1, wherein the artificial intelligence processing die is configured to transmit a response signal to the processor in response to realizing the configuration in the artificial intelligence processing die based on the selected input.

3. The system according to claim 1 or 2, wherein the artificial intelligence processing die is further configured to send an error message to the processor in response to the detection of an error when configuring the neural network processing operation based on the selected input.

4. Configuring the neural network processing operation is The artificial intelligence processing die receives configuration data indicating the configuration for realizing the neural network layer, The system according to claim 3, further comprising determining an inter-die communication path between the artificial intelligence processing die and other artificial intelligence processing dies of the system based on the configuration data.

5. The system according to any one of claims 1 to 4, wherein the processor selects the input representing the configuration based on a unique identifier associated with the artificial intelligence processing die.

6. The system according to any one of claims 1 to 5, wherein the processor transmits the selected input to the artificial intelligence processing die via the host interface unit of the artificial intelligence processing die.

7. The processor is Periodically determine whether the configuration data for the artificial intelligence processing die has been updated. The system according to any one of claims 1 to 6, wherein the system is configured to transmit the updated configuration data to the artificial intelligence processing die in response to determining that the configuration data for the artificial intelligence processing die has been updated.

8. i) The configuration data is stored in the memory of the host computing device, ii) The system according to claim 7, wherein each of the processor and the artificial intelligence processing die is configured to read the configuration data from the memory of the host computing device.

9. The system according to claim 8, wherein the system is housed within the host computing device.

10. The system according to claim 9, wherein the artificial intelligence processing die reads the configuration data from the memory of the host computing device based on a control signal command transmitted to the artificial intelligence processing die by the processor.

11. A method for processing a neural network computation task, A processor coupled to an artificial intelligence processing die selects inputs that represent a configuration used to implement a neural network layer in the artificial intelligence processing die, The processor transmits the selected input to the artificial intelligence processing die, A method comprising configuring the neural network processing operation for the neural network layer in the artificial intelligence processing die according to the configuration represented by the selected input.

12. The method according to claim 11, further comprising the artificial intelligence processing die transmitting a response signal to the processor in response to the artificial intelligence processing die realizing the configuration based on the selected input.

13. The method according to claim 11 or 12, further comprising the artificial intelligence processing die sending an error message to the processor in response to the detection of an error when configuring the neural network processing operation based on the selected input.

14. Configuring the neural network processing operation is The artificial intelligence processing die receives configuration data indicating the configuration for realizing the neural network layer, The method according to claim 13, further comprising determining an inter-die communication path between the artificial intelligence processing die and other artificial intelligence processing dies based on the configuration data.

15. The method according to any one of claims 11 to 14, further comprising the processor selecting the input representing the configuration based on a unique identifier associated with the artificial intelligence processing die.

16. The method according to any one of claims 11 to 15, further comprising the processor transmitting the selected input to the artificial intelligence processing die via the host interface unit of the artificial intelligence processing die.

17. The processor periodically determines whether the configuration data for the artificial intelligence processing die has been updated, The method according to any one of claims 11 to 16, further comprising the processor transmitting the updated configuration data to the artificial intelligence processing die when the processor determines that the configuration data for the artificial intelligence processing die has been updated.

18. i) The configuration data is stored in the memory of the host computing device, ii) The method according to claim 17, wherein each of the processor and the artificial intelligence processing die is configured to read the configuration data from the memory of the host computing device.

19. The method according to claim 18, wherein the artificial intelligence processing die reads the configuration data from the memory of the host computing device based on a control signal command transmitted to the artificial intelligence processing die by the processor.

Images