Artificial intelligence chip and integrated device

Abstract

The application relates to the technical field of artificial intelligence hardware, and provides an artificial intelligence chip and an integrated device, wherein the artificial intelligence chip comprises a plurality of computing dies which are connected in series; in a first state, the plurality of computing dies are configured to run as a first device; in a second state, the plurality of computing dies are configured to run as a plurality of second devices respectively; and the switching between the first state and the second state is determined based on memory access ranges of the plurality of computing dies. In the artificial intelligence chip and the integrated device provided by the application, the memory access ranges of the plurality of computing dies are configured, so that the artificial intelligence chip can run in the first state or the second state, the artificial intelligence chip can be flexibly adapted to different algorithms, model requirements and different application scenarios through state switching while the overall computing power is improved through the running of the plurality of computing dies, and the computing power and application flexibility of the artificial intelligence chip are doubly optimized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence hardware technology, and in particular to an artificial intelligence chip and integrated device. Background Technology

[0002] As the core hardware supporting the development of artificial intelligence (AI) technology, the performance of AI chips is key to determining whether AI algorithms can run efficiently and be commercially deployed.

[0003] To meet the ever-increasing demand for computing power, current AI chip designs tend to employ highly specialized hardware architectures to pursue performance. However, highly specialized AI chips have relatively rigid hardware architectures and poor application flexibility. Especially for rapidly evolving AI algorithms and application scenarios, this lack of flexibility directly limits the application scope of such AI chips. Summary of the Invention

[0004] This invention provides an artificial intelligence chip and an integrated device to address the shortcomings of related technologies in that it is difficult to balance computing power and flexibility.

[0005] This invention provides an artificial intelligence chip, comprising multiple computing dies connected in series:

[0006] In the first state, the plurality of computing dies are configured to operate as a first device;

[0007] In the second state, the plurality of computing dies are configured to operate as a plurality of second devices respectively;

[0008] The switching between the first state and the second state is determined based on the memory access range of the plurality of computing dies.

[0009] This invention provides an artificial intelligence chip, wherein in the first state, the memory access range of each computing die includes the memory of all computing dies;

[0010] In the second state, the memory access range of each computing die includes the memory of the computing die in the second device to which it belongs.

[0011] This invention provides an artificial intelligence chip, wherein the computing die includes a first interface and a second interface, and the bandwidth of the first interface is greater than the bandwidth of the second interface;

[0012] Among the plurality of computing dies, two adjacent computing dies are connected via the first interface or via the second interface.

[0013] The present invention provides an artificial intelligence chip, wherein in the second state, two computing dies connected by the first interface are configured as the same second device, and two computing dies connected by the second interface are configured as different second devices.

[0014] The present invention provides an artificial intelligence chip, which further includes an input / output die, wherein the input / output die and the computing die of the serially connected connection terminal are connected through the second interface.

[0015] The present invention provides an artificial intelligence chip, which further includes an external storage control unit. The external storage control unit is connected to a computing die of a serially connected connection terminal through a second interface. The external storage control unit is used to connect to external memory, which is disposed outside the artificial intelligence chip.

[0016] The present invention provides an artificial intelligence chip, wherein each of the computing dies includes a replaceable component; the replaceable component of one of the computing dies is enabled, while the replaceable components of the other computing dies are disabled.

[0017] The present invention provides an artificial intelligence chip, wherein the plurality of second devices include a master device and a slave device.

[0018] This invention provides an artificial intelligence chip in which multiple computing dies are connected based on a common die-interconnect technology.

[0019] The present invention also provides an integrated device, including a substrate and at least one artificial intelligence chip as described above disposed on the substrate.

[0020] The artificial intelligence chip and integrated device provided by this invention can switch the operation of the artificial intelligence chip in a first state or a second state by configuring the memory access range of multiple computing dies. Furthermore, in the second state, the number of second devices and the number of computing dies contained in the second devices can also be adjusted by configuring the memory access range of the multiple computing dies in the artificial intelligence chip. This allows the artificial intelligence chip to improve its overall computing power by operating multiple computing dies, while flexibly adapting to different algorithm and model requirements and different application scenarios through state switching. The computing power and application flexibility of the artificial intelligence chip are thus optimized in two ways. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is one of the structural schematic diagrams of the artificial intelligence chip provided by the present invention.

[0023] Figure 2 This is a schematic diagram of the serial connection of the computing die provided by the present invention.

[0024] Figure 3 This is one of the structural schematic diagrams of the input / output die provided by the present invention.

[0025] Figure 4 This is the second schematic diagram of the structure of the artificial intelligence chip provided by the present invention.

[0026] Figure 5 This is the third schematic diagram of the structure of the artificial intelligence chip provided by the present invention.

[0027] Figure 6 This is a schematic diagram of the structure of the computational die provided by the present invention.

[0028] Figure 7 This is a schematic diagram of the integrated device provided by the present invention.

[0029] Figure 8 This is the second schematic diagram of the input / output die provided by the present invention.

[0030] Figure label:

[0031] 1: Artificial intelligence chip; 10: Computing die; 11: First device;

[0032] 12: Second device; 101: First interface; 102: Second interface;

[0033] 20: Input / output die; 201: UCIe interface; 202: PCIe interface;

[0034] 203: RDMA interface; 204: P2Plink interface; 205: CP;

[0035] 30: External storage control unit; 40: External memory; 103: On-chip router;

[0036] 104: SPC; 105: Cache; 106: Storage Control Unit;

[0037] 107: On-chip memory; 2: Substrate. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] With the rapid development of artificial intelligence technology, especially the rise of applications such as deep learning and large-scale language models, the complexity of artificial intelligence algorithms is constantly increasing and the amount of data is exploding. Various artificial intelligence applications are placing increasingly higher demands on the computing power of underlying hardware.

[0040] As the core hardware supporting the development of artificial intelligence technology, the performance of artificial intelligence chips is the key to determining whether artificial intelligence algorithms can run efficiently and be commercially deployed.

[0041] To meet the ever-increasing demand for computing power, current AI chip designs tend to employ highly specialized hardware architectures to pursue performance. However, AI technology is still undergoing rapid iteration and evolution, with new algorithms and models constantly emerging, and the application scenarios of AI technology are becoming increasingly diversified. These highly specialized AI chips, with their relatively fixed hardware architectures, suffer from poor application flexibility. When the algorithms, models, or application scenarios they need to process change, these AI chips may fail to provide effective support, leading to a sharp drop in computational efficiency or even complete incompatibility.

[0042] This lack of application flexibility not only limits the application scope of AI chips but also increases the cost of product iteration and maintenance. Therefore, how to improve the application flexibility of AI chips while ensuring high computing power, so that they can meet the high-performance computing needs of current mainstream applications and be compatible with and adaptable to new algorithms and applications that may emerge in the future, is a technical problem that urgently needs to be solved in this field.

[0043] To address the aforementioned problems, this invention provides an artificial intelligence chip. This artificial intelligence chip can be at least one of a GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Unit), TPU (Tensor Processing Unit), NPU (Neural network Processing Unit), DPU (Deep learning Processing Unit), and APU (Accelerated Processing Unit).

[0044] Figure 1 This is one of the structural schematic diagrams of the artificial intelligence chip provided by the present invention, such as... Figure 1 As shown, the artificial intelligence chip 1 includes multiple computing dies 10, which are connected in series:

[0045] In the first state, the plurality of computing dies 10 are configured to operate as a first device 11;

[0046] In the second state, the plurality of computing dies 10 are configured to operate as a plurality of second devices 12 respectively;

[0047] The switching between the first state and the second state is determined based on the memory access range of the plurality of computing dies 10.

[0048] Specifically, a die refers to a silicon wafer containing a complete integrated circuit and having a single function. In this embodiment of the invention, the artificial intelligence chip 1 includes a compute die 10. The compute die 10 is the die responsible for computing tasks, and may include processors such as SPC (Streaming Processing Cluster) and caches.

[0049] The number of computing dies 10 included in the artificial intelligence chip 1 can be multiple, and the multiple computing dies 10 have the same structure and the same function. For example, the artificial intelligence chip 1 can be provided with 2 computing dies 10, or 3 or 4 computing dies 10, etc. The embodiments of the present invention do not specifically limit the number of computing dies 10.

[0050] In the case of multiple computing dies 10 in the artificial intelligence chip 1, these dies 10 can be connected serially to enable data transmission between them. Understandably, in a serial connection structure, any two computing dies 10 can exchange data. Specifically, if any two computing dies 10 are directly connected, they can directly exchange data; if any two computing dies 10 are connected through one or more other computing dies 10, they can exchange data through those other computing dies 10.

[0051] It is understandable that by serially connecting multiple computing dies 10 in the artificial intelligence chip 1, the computing power of the computing dies 10 can be superimposed, thereby significantly improving the overall computing power of the artificial intelligence chip 1.

[0052] Based on this, two states can be configured for the operation of AI chip 1, which are referred to here as the first state and the second state, respectively.

[0053] In this context, the first state refers to treating all the computing dies 10 in the AI ​​chip 1 as a whole, and denoteing the whole composed of all the computing dies 10 as a first device 11. The first device 11 can be understood as a virtual computing device, and its computing power is the sum of the computing power of all the computing dies 10. When the AI ​​chip 1 operates in the first state, the first device 11 operates, and all the computing dies 10 contained within the first device 11 cooperate with each other to achieve the same computing task.

[0054] The second state refers to dividing all the computing dies 10 in the AI ​​chip 1 into multiple parts, each operating independently. These multiple parts are referred to as multiple second devices 12. A second device 12 can be understood as a virtual computing device, and its computing power is the sum of the computing power of the computing dies 10 it contains. In the second state, each second device 12 can operate independently, and based on the connection relationships between different second devices 12, data can be transmitted between them to achieve collaborative operation.

[0055] Here, in the AI ​​chip 1, the number of second devices 12 can be multiple, and the number of second devices 12 is less than or equal to the number of computing dies 10. Furthermore, each second device 12 may contain one or more computing dies 10, and the number of computing dies 10 contained in different second devices 12 can be the same or different. For example, if the AI ​​chip 1 contains 4 computing dies 10, it can be divided into 2 second devices 12, each second device 12 including 2 adjacent computing dies 10; or, if the AI ​​chip 1 contains 2 computing dies 10, it can be divided into 2 second devices 12, each second device 12 including one computing die 10; or, if the AI ​​chip 1 contains 4 computing dies 10, it can be divided into 3 second devices 12, one second device 12 including two adjacent computing dies 10, and the other two second devices 12 each including one computing die 10.

[0056] The first and second states described above can be determined by configuring the memory access ranges of multiple computing dies 10. For any given computing die 10, the memory access range refers to the range of memory accessible by a memory access initiated by that computing die 10. This memory access range can include the memory of the computing die 10 itself, and can also include the memory of other computing dies 10. It is understood that computing dies 10 belonging to the same device can access each other's memory. That is, in the first state, each computing die 10 can access the memory of all computing dies 10, thus all computing dies 10 can be considered as a single first device 11; in the second state, each computing die 10 can only access a portion of the memory of other computing dies 10, thus computing dies 10 that can access each other's memory are considered the same second device 12, and computing dies 10 that cannot access each other's memory are considered different second devices 12.

[0057] In the artificial intelligence chip provided in this embodiment of the invention, by configuring the memory access range of multiple computing dies, the artificial intelligence chip can be switched to run in a first state or a second state. Furthermore, in the second state, by configuring the memory access range of the multiple computing dies in the artificial intelligence chip, the number of second devices and the number of computing dies contained within the second devices can also be adjusted. This allows the artificial intelligence chip to improve its overall computing power by running multiple computing dies, while also flexibly adapting to different algorithms, model requirements, and application scenarios through state switching. The computing power and application flexibility of the artificial intelligence chip are thus dually optimized.

[0058] Based on the above embodiments, in the first state, the memory access range of each computing die includes the memory of all computing dies;

[0059] In the second state, the memory access range of each computing die includes the memory of the computing die in the second device to which it belongs.

[0060] Specifically, in artificial intelligence chips, the state switching of the chip can be achieved by configuring the memory access range of each computing die.

[0061] In the first state, the memory access range of each computing die can be set to the memory of all computing dies. Therefore, for each computing die in the AI ​​chip, its memory access range includes the memory of all computing dies in the AI ​​chip. Thus, each computing die can access the memory of all computing dies. During the operation of the first device, which treats all computing dies as a whole, all computing dies share memory, significantly improving the computing performance of the first device integrating all computing dies.

[0062] For example, consider an AI chip containing four compute dies, denoted as Compute die0, Compute die1, Compute die2, and Compute die3. In the first state, the memory access range of Compute die0 can include the memory of Compute die0, Compute die1, Compute die2, and Compute die3. That is, the physical address of a memory access request originating from Compute die0 can include the physical addresses of Compute die0, Compute die1, Compute die2, and Compute die3. The memory access range of Compute die1, Compute die2, and Compute die3 is the same as that of Compute die0.

[0063] For the second state, the memory range of each computing die can be set to the memory of every computing die under the second device to which it belongs. Therefore, for each computing die in the AI ​​chip, the memory access range of the computing die includes the memory of every computing die under its own second device. Thus, each computing die can access the memory of every computing die under its own second device, but cannot access the memory of computing dies under other second devices. This achieves memory isolation between different second devices, while computing dies within the same second device share memory.

[0064] For example, consider an AI chip containing four compute dies, denoted as Compute die0, Compute die1, Compute die2, and Compute die3. In a second state, assuming Compute die0 and Compute die1 are configured as one second device, and Compute die2 and Compute die3 as another second device, then the memory access range of Compute die0 and Compute die1 includes the memory of Compute die0 and Compute die1. That is, the physical address of a memory access request originating from Compute die0 and Compute die1 can include physical addresses within Compute die0 and Compute die1; similarly, the memory access range of Compute die2 and Compute die3 includes the memory of Compute die2 and Compute die3. That is, the physical address of a memory access request originating from Compute die2 and Compute die3 can include physical addresses within Compute die2 and Compute die3.

[0065] In the artificial intelligence chip provided in this embodiment of the invention, the configuration of the memory access range enables the configuration of the operating state of the artificial intelligence chip, thereby enabling the artificial intelligence chip to operate in a state that adapts to the algorithm, model requirements, and application scenarios.

[0066] Based on any of the above embodiments, the computing die includes a first interface and a second interface, wherein the bandwidth of the first interface is greater than the bandwidth of the second interface;

[0067] Among the plurality of computing dies, two adjacent computing dies are connected via the first interface or via the second interface.

[0068] Specifically, the computing die may include a first interface and a second interface, both of which are data transmission interfaces. Through the first and second interfaces, the computing die can connect to other dies within the artificial intelligence chip, such as connecting to other computing dies, or connecting to input / output dies, storage control unit dies, etc. The computing die can also connect to other devices outside the artificial intelligence chip, such as connecting to computing dies within other artificial intelligence chips; however, this embodiment of the invention does not specifically limit this connection.

[0069] For a given computing die, the difference between the first interface and the second interface is that the bandwidth of the first interface is greater than that of the second interface. This bandwidth difference can be achieved by configuring different data transmission protocols for the first and second interfaces, or by setting different numbers of transmission ports for the first and second interfaces. For example, a larger number of ports can be set for the first interface, and a smaller number for the second interface, thus making the bandwidth of the first interface greater than that of the second interface.

[0070] Understandably, providing two interfaces with different bandwidths for the computing die offers multiple possible options for its operation in a serial connection within an AI chip, thus improving the application flexibility of the AI ​​chip. Furthermore, the lower-bandwidth second interface requires less design area, which is more beneficial for reducing the die area and improving the integration of the AI ​​chip compared to designing two high-bandwidth interfaces.

[0071] Therefore, in a serial connection structure of multiple computing dies, two adjacent computing dies can be connected via a first interface or a second interface. Furthermore, in this serial connection structure, computing dies at either end of the structure only need to be connected to adjacent computing dies via either the first or second interface, while computing dies in the middle of the structure need to be connected to one adjacent computing die via the first interface at one end and to another adjacent computing die via the second interface at the other end.

[0072] For example, Figure 2 This is a schematic diagram of the serial connection of the computing die provided by the present invention. Figure 2 There are four serially connected computing dies 10. In order from top to bottom, the first computing die 10 is connected to the first interface 101 of the second computing die 10 through the first interface 101, the second computing die 10 is connected to the second interface 102 of the third computing die 10 through the second interface 102, and the third computing die 10 is connected to the first interface 101 of the fourth computing die 10 through the first interface 101.

[0073] Based on any of the above embodiments, in the second state, two computing dies connected by the first interface are configured as the same second device, and two computing dies connected by the second interface are configured as different second devices.

[0074] Specifically, since the bandwidth of the first interface is greater than that of the second interface, the data transmission efficiency between two computing dies connected via the first interface is obviously greater than that between two computing dies connected via the second interface. Therefore, in the second state, the two computing dies connected via the first interface can be configured as the same second device, and the two computing dies connected via the second interface can be configured as different second devices. In this way, the computing dies within the second device are interconnected via the first interface, and data transmission within the second device can be achieved with higher bandwidth, thus improving the computing performance of the second device.

[0075] For example, in Figure 2 In the second state, following a top-down order, the first and second computational dies 10 can be configured as one second device, and the third and fourth computational dies 10 can be configured as another second device. Thus, in Figure 2 Inside the two second devices, the computing dies can be connected via a first interface with higher bandwidth.

[0076] Based on any of the above embodiments, the artificial intelligence chip further includes an input / output die, which is connected to the computing die of the serially connected connection terminal through the second interface.

[0077] Specifically, an input / output die (SPC) can also be included in the artificial intelligence chip. The SPC is responsible for data input / output and can include at least one type of external interface. This external interface enables connection between the artificial intelligence chip and external devices, thereby facilitating data transmission between them. The external interface can be a PCIe (Peripheral Component Interconnect Express) interface, an RDMA (Remote Direct Memory Access) interface, or an interface corresponding to a manufacturer's proprietary connection method. This embodiment of the invention does not impose specific limitations on this. For example, the artificial intelligence chip can connect to a CPU (Central Processing Unit) via the PCIe interface on the SPC, or connect to other artificial intelligence chips via the manufacturer's proprietary connection method interface on the SPC, or connect to a network card via RDMA, thereby enabling the artificial intelligence chip to connect to a large-scale cluster.

[0078] Furthermore, the input / output dies are provided with interfaces corresponding to the second interfaces of the computing dies, thereby enabling connection with the second interfaces of the computing dies. It is understood that in an AI chip, multiple computing dies are connected serially. In this serial connection structure, the computing dies at both ends of the serial connection structure, while connected to adjacent computing dies via the first interface, can also use the second interface for connection with other dies. Thus, input / output dies can be connected to computing dies via the second interface, and the computing die connected to the input / output dies is the computing die at the connection end, specifically one of the computing dies at both ends of the serial connection structure. Additionally, in the case of two input / output dies, the two input / output dies can also be connected one-to-one to the two computing dies at both ends of the serial connection structure.

[0079] For example, the second interface of the computing die is the Universal Chiplet Interconnect Express (UCIe) interface, so that input / output dies can also be connected to the computing die through Universal Chiplet Interconnect Express.

[0080] For example, Figure 3 This is one of the structural schematic diagrams of the input / output die provided by the present invention, such as... Figure 3 As shown, the input / output die 20 includes an external interface for connecting to external devices via a UCIe interface 201 connected to the computing die, and a CP (Command Processor) 205. The CP 205 manages and schedules input / output resources; the external interface may include a PCIe interface 202, an RDMA interface 203, and a P2Plink (Point to Point Link) interface 204.

[0081] For example, Figure 4 This is the second structural schematic diagram of the artificial intelligence chip provided by the present invention, as shown below. Figure 4 As shown, in the artificial intelligence chip 1, there are four serially connected computing dies 10, of which two computing dies 10 at both ends of the serial connection structure are connected to the input / output dies 20 respectively.

[0082] Based on any of the above embodiments, the artificial intelligence chip further includes an external storage control unit, which is connected to the computing die of the serially connected connection terminal through the second interface. The external storage control unit is used to connect to external memory, which is disposed outside the artificial intelligence chip.

[0083] Specifically, in this embodiment of the invention, memory located outside the artificial intelligence chip and independent of the AI ​​chip is referred to as external memory. External memory typically has a large capacity; for example, external memory may be LPDDR (Low Power Double Data Rate). The number of external memory modules can be one or more, and the type of external memory can be one or more. External memory typically has a large size and therefore is usually located outside the AI ​​chip.

[0084] The external storage control unit is an MCU (Memory Control Unit) connected to external memory. It can also be called a memory controller. The MCU is designed to translate memory access requests issued by the computing units in the computing die into a sequence of instructions that the memory can understand, and to manage the data flow to the memory.

[0085] The external storage control unit is equipped with an interface corresponding to the second interface of the computing die, thereby enabling connection with the second interface of the computing die. It is understood that in an AI chip, multiple computing dies are connected serially. In this serial connection structure, the computing dies at both ends of the serial connection structure, while connected to adjacent computing dies via the first interface, can also use the second interface for connection with other dies. Therefore, the external storage control unit can be connected to the computing die via the second interface, and the computing die connected to the external storage control unit is the computing die at the connection end, specifically one of the computing dies at both ends of the serial connection structure. Furthermore, in the case of two external storage control units, both external storage control units can also be connected one-to-one to the two computing dies at both ends of the serial connection structure.

[0086] For example, Figure 5 This is the third schematic diagram of the structure of the artificial intelligence chip provided by the present invention, as shown below. Figure 5 As shown, there are two computing dies 10 in the artificial intelligence chip 1 connected through a first interface 101. The computing die 10 arranged above is connected to an external storage control unit 30 through a second interface 102, and is connected to an external memory 40 located outside the artificial intelligence chip 1 through the external storage control unit 30. The computing die 10 arranged below is connected to an input / output die 20 through a second interface 102.

[0087] Based on any of the above embodiments, each of the computing dies includes a replaceable component; the replaceable component of one of the plurality of computing dies is enabled, while the replaceable components of the other computing dies are disabled.

[0088] Specifically, replaceable components refer to components that must run on any one of the compute dies in an AI chip. These could be, for example, a command processor (CP) or a PCIe interface. Replaceable components can be configured on each compute die of the AI ​​chip. During operation, only the replaceable component on one compute die needs to be enabled, while the replaceable components on other compute dies are disabled. Thus, the AI ​​chip runs all replaceable components on a single compute die.

[0089] In this embodiment of the invention, a replaceable component is provided on each computing die of the artificial intelligence chip. This redundant design ensures that if a replaceable component in any computing die fails, its function can be replaced by a replaceable component that is running normally on another computing die, thereby ensuring the operational reliability of the replaceable component.

[0090] Based on any of the above embodiments, the plurality of second devices include master devices and slave devices.

[0091] Specifically, in the second state, multiple computing dies can be divided into multiple second devices, with one second device serving as the master device and the remaining second devices serving as slave devices, thus forming a master-slave structure. For example, the master device is responsible for receiving computing tasks, decomposing the computing tasks into multiple subtasks that can be executed in parallel and distributing them to the slave devices, and receiving the computing results obtained by the slave devices after the computing is completed, and aggregating all computing results into the final output.

[0092] In this embodiment of the invention, by dividing multiple second devices into master and slave devices in the second state, parallel computing can be realized in the second state, thereby optimizing the performance of the artificial intelligence device in the second state and further improving the scalability and application flexibility of the artificial intelligence chip.

[0093] Based on any of the above embodiments, the plurality of computing dies are connected based on a common die-interconnect technology.

[0094] Specifically, Universal Interconnect for Chips (UCIe) is a high-speed interconnect protocol standard. In artificial intelligence chips, UCIe technology can be used to achieve serial connections between multiple computing dies.

[0095] Furthermore, for cases where AI chips also include input / output dies, the input / output dies can also be connected to the computing dies via UCIe technology; and for cases where AI chips also include external storage control units, the external storage control units can also be connected to the computing dies as a type of die via UCIe technology.

[0096] In this embodiment of the invention, in the artificial intelligence chip, the connection between various dies can be realized through UCIe, thereby realizing the integration of multiple dies and the relative decoupling between multiple dies, making the die-level iteration in the artificial intelligence chip more flexible.

[0097] Based on any of the above embodiments Figure 6 This is a schematic diagram of the structure of the computational die provided by the present invention, as shown below. Figure 6 As shown, in an artificial intelligence chip, each computing die 10 can be represented by the following structure:

[0098] The computing die 10 may include a first interface 101, a second interface 102, an on-chip router 103 (also known as an MNode), a stream processor cluster SPC 104, a cache 105 (specifically, a secondary cache L2 cache), a storage control unit 106, and on-chip memory 107.

[0099] Both the first interface 101 and the second interface 102 can be UCIe interfaces. Figure 6 The circles on the UCIe interface indicate the ports of the UCIe interface. The number of ports on the first interface 101 is greater than the number of ports on the second interface 102.

[0100] Each SPC104 contains 16 EUs (Execution Units), and four EUs can be configured into one CU (Compute Unit). Each EU contains 16 general-purpose streaming processors and a tensor engine TCore. The SPC104 can handle the computational tasks of the compute die, and one or more SPC104s can be included on the compute die 10. Furthermore, one or more caches 105 can be included on the compute die 10.

[0101] The on-chip router 103 can be connected to each SPC 104 and each cache 105 in the computing chip 10. Furthermore, the on-chip router 103 can also be connected to the first interface 101 and the second interface 102 in the computing chip 10. Thus, the on-chip router 103 can coordinate and manage the data interaction between each SPC 104 and cache 105 in the computing chip 10, and can also handle data interaction with devices outside the computing chip 10. The diamond-shaped marker between the SPC 104 and the on-chip router 103 is called Pagen (Physical Address Generation). Pagen is used to translate the virtual address output by the SPC 104 into a physical address, facilitating routing by the on-chip router 103.

[0102] The cache 105 in the compute die 10 is also connected to the storage control unit 106, which is used to connect to the on-chip memory 107, thereby enabling memory access to the on-chip memory 107. For example, the on-chip memory 107 can be HBM (High Bandwidth Memory).

[0103] Based on any of the above embodiments, the present invention provides an integrated device, including a substrate and at least one artificial intelligence chip as described above disposed on the substrate.

[0104] The substrate is the physical layer that carries and connects the AI ​​chip; it can be a printed circuit board, a ceramic substrate, or a silicon interposer. The substrate contains conductive layers and wiring to enable the AI ​​chip to connect with other devices deployed on the substrate.

[0105] Therefore, in the integrated device, by configuring the memory access range of multiple computing dies of the AI ​​chip, the operation of the AI ​​chip in the first or second state can be switched. Furthermore, in the second state, by configuring the memory access range of multiple computing dies in the AI ​​chip, the number of second devices and the number of computing dies contained within the second devices can also be adjusted. This allows the AI ​​chip to improve its overall computing power by operating multiple computing dies, while flexibly adapting to different algorithms, model requirements, and application scenarios through state switching. The computing power and application flexibility of the AI ​​chip are thus dually optimized.

[0106] For example, Figure 7 This is a schematic diagram of the integrated device provided by the present invention. Figure 7 As shown, the integrated device includes a substrate 2, an artificial intelligence chip 1 and an external memory 40 deployed on the substrate 2.

[0107] The artificial intelligence chip 1 includes two computing dies 10, one input / output die 20, and one external storage control unit 30. Each of the two computing dies 10 has a first interface 101 and a second interface 102, both of which are UCIe interfaces. The two computing dies 10 are connected to each other through their respective first interfaces 101, and one computing die 10 is connected to the input / output die 20 through the second interface 102, while the other computing die 10 is connected to the external storage control unit 30 through the second interface 102.

[0108] The number of UCIe links between computing dies 10 implemented based on the first interface 101 can be 6-10 times the number of UCIe links between computing dies 10 and input / output dies 20 implemented based on the second interface 102, or the number of UCIe links between computing dies 10 and external storage control unit 30 implemented based on the second interface 102. This can achieve bandwidth differentiation and thus save chip area.

[0109] The two computing dies 10 have the same structure, each including a first interface 101, a second interface 102, an on-chip router 103, a stream processor cluster (SPC) 104, a cache 105, a storage control unit 106, on-chip memory 107, as well as a CP 205 and a PCIe interface 202. The CP 205 and PCIe interface 202 are considered replaceable components within the computing dies 10. Enabling the use of replaceable components in only one of the two computing dies 10 is sufficient to ensure the normal operation of the CP 205 and PCIe interface 202 on both computing dies 10.

[0110] The external storage control unit 30 can be connected to the external memory 40 deployed on the substrate 2. Thus, the AI ​​chip 1 can access memory not only the on-chip memory 107 built into the computing die 10, but also the external memory 40 outside the AI ​​chip 1 via the external storage control unit 30. Furthermore, a cache (i.e., L2 cache shown in the figure) can be deployed on the external storage control unit 30.

[0111] In addition, since the CP205 and PCIe interface 202 are already deployed inside the computing die 10, it is not necessary to deploy CP205 and PCIe interface 202 separately in the input / output die. Figure 8 This is a second schematic diagram of the input / output die provided by the present invention, as shown below. Figure 8 As shown, the input / output die 20 can be connected to the computing die via the UCIe interface 201, and can also be connected to external devices via the RDMA interface 203 and P2Plink connection.

[0112] In the integrated device provided in this embodiment of the invention, the computing power and application flexibility of the artificial intelligence chip are dually optimized through the connection of multiple computing dies. Furthermore, the memory capacity of the artificial intelligence chip is expanded by connecting the computing dies to the external storage control unit, and by connecting the external storage control unit to external memory. In addition, this integrated device also supports the filtering and function masking of CP and PCIe interfaces.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An artificial intelligence chip, characterized in that, It includes multiple computing dies, which are connected in series: In the first state, the plurality of computing dies are configured to operate as a first device; In the second state, the plurality of computing dies are configured to run as a plurality of second devices, and data is transmitted between the plurality of second devices to achieve collaborative operation of different devices; The switching between the first state and the second state is determined based on the memory access range of the plurality of computing dies. The first state refers to treating all computing dies in the artificial intelligence chip as a whole, and the whole composed of all computing dies is denoted as a first device. The second state refers to dividing all computing dies in the artificial intelligence chip into multiple parts, each of which operates independently.

2. The artificial intelligence chip according to claim 1, characterized in that, In the first state, the memory access range of each computing die includes the memory of all computing dies; In the second state, the memory access range of each computing die includes the memory of the computing die in the second device to which it belongs.

3. The artificial intelligence chip according to claim 1, characterized in that, The computing die includes a first interface and a second interface, wherein the bandwidth of the first interface is greater than the bandwidth of the second interface; Among the plurality of computing dies, two adjacent computing dies are connected via the first interface or via the second interface.

4. The artificial intelligence chip according to claim 3, characterized in that, In the second state, two computing dies connected by the first interface are configured as the same second device, and two computing dies connected by the second interface are configured as different second devices.

5. The artificial intelligence chip according to claim 3, characterized in that, It also includes input / output dies, which are connected to the computing dies of the serially connected connection end through the second interface.

6. The artificial intelligence chip according to claim 3, characterized in that, It also includes an external storage control unit, which is connected to the computing die of the serially connected connection terminal through the second interface. The external storage control unit is used to connect to external memory, which is located outside the artificial intelligence chip.

7. The artificial intelligence chip according to any one of claims 1 to 6, characterized in that, Each of the computing dies includes a replaceable component; the replaceable component of one of the computing dies is enabled, while the replaceable components of the other computing dies are disabled.

8. The artificial intelligence chip according to any one of claims 1 to 6, characterized in that, The plurality of second devices includes master devices and slave devices.

9. The artificial intelligence chip according to any one of claims 1 to 6, characterized in that, The multiple computing dies are interconnected based on a common die-interconnect technology.

10. An integrated device, characterized in that, It includes a substrate, and at least one artificial intelligence chip disposed on the substrate as described in any one of claims 1 to 9.

Images