Recomposable reasoning platform

Abstract

A reconfigurable inference platform implemented in a system-in-package configuration is provided. The system in the reconfigurable inference platform includes a processor chip, a first memory chip storing a set of executable models for inference engines, and a second memory chip storing weights for a selected executable model. The processor chip has a running processor core, an accelerator core, and a processor-memory interface exposed on a chip-to-chip bonding surface. The first memory chip is, for example, a three-dimensional NAND flash memory. The second memory chip can include a non-volatile random access memory, such as a phase change memory. A direct vertical connection structure, such as a via-to-via connection structure, is provided between the processor chip and the second memory chip.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to computing platforms for inference operations using artificial intelligence models, and models generated using machine learning; and more specifically to such platforms suitable for use in edge devices. BACKGROUND

[0002] Systems that execute computing models developed using machine learning, including artificial intelligence models, involve the use of large arrays of coefficients to perform a large number of arithmetic operations across multiple input arrays. The coefficients are often referred to as weights. In platforms that execute these models, off-chip memory access can limit power and performance. Due to the size of the arrays of coefficients used in these models, on-chip memory can be insufficient, especially in systems that use multiple models.

[0003] There is a need to provide platforms for inference operations that address these issues.

[0004] SUMMARY

[0005] A reconfigurable inference platform suitable for implementation in a system in package (SiP) configuration is described. The reconfigurable inference platform described herein can include a processor chip, a first memory chip suitable for storing a plurality of arrays of weights, and a second memory chip suitable for storing a plurality of sets of executable models. The platform can be implemented as a multi-chip module in a single package. The package can be mounted on a circuit board or other type of substrate, and connected to sensors and other elements that can generate data consumed by the executable models and can consume data generated by the executable models.

[0006] In implementations of the reconfigurable inference platform, a processor chip can include a runtime processor core, an accelerator core, and a processor-memory interface exposed on a chip-to-chip bonding surface of the processor chip. In implementations of the platform, a first memory chip can include non-volatile high-capacity memory, such as three-dimensional NAND flash memory. The first memory chip can store a set of executable models of an inference engine, where each executable model includes a set of weights for executing the model, and in some cases also a computation graph for the inference engine. A second memory chip can store at least the set of weights of a selected executable model. The second memory chip can include non-volatile random access memory, such as phase change memory. The second memory chip can include a memory-processor interface exposed on a surface of the second memory chip and complementary to the processor-memory interface on the processor chip. A plurality of direct vertical connections, such as via-to-via connections, are provided between the processor-memory interface and the memory-processor interface, which can enable data to be transferred between the chips with low power, high throughput, and low latency transfer to support execution of the selected model.

[0007] In examples described herein, the processor chip and the second memory chip are stacked and disposed on an interposer. The first memory chip is electrically disposed on the interposer, which includes interconnection wiring that forms at least a portion of a data path between the first memory chip and the second memory chip. In addition to the processor-memory interface, the processor chip can include an input / output interface, and the data path can include connections from the interconnection wiring of the interposer to the input / output interface of the processor chip.

[0008] In the examples described herein, a processor chip can access an instruction memory, which can be included on the processor chip or accessible in off-chip storage, to store instructions to run a program. The program can include selecting an executable model from a set of executable models stored in a first memory chip, loading a computation graph for the selected model (including configuring an accelerator core on the processor chip), transferring a set of weights for the selected model to a second memory chip, and executing the selected model. In addition, the program can include changing the model in response to a control event in the field. Thus, the program can include changing the selected model to a different model in the set of executable models, loading a computation graph for the different model (including configuring the accelerator core), transferring a set of weights for the different model to the second memory chip, and executing the different model.

[0009] An example of a reconfigurable inference method is described, including providing a processor chip including a processing core, an accelerator core, on-chip memory, and a processor-memory interface exposed on a chip-to-chip bonding surface of the processor chip; storing a set of executable models for an inference engine of a model implemented in machine learning in a first memory chip, the first memory chip accessible to the processor chip, each model including a set of weights for executing the model; in response to a control event, selecting an executable model from the set of executable models stored in the first memory chip; loading a computation graph for the selected model, including configuring the accelerator core; transferring the set of weights for the selected executable model from the first memory chip to a second memory chip, where the second memory chip includes a memory-processor interface disposed on a surface of the second memory chip and complementary to the processor-memory interface; and executing the selected executable model using a plurality of direct vertical connections between the processor-memory interface and the memory-processor interface.

[0010] Other aspects and benefits of the present application will become apparent from the following drawings, descriptions, and appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A multi-chip module including a reconfigurable inference platform as described herein is illustrated;

[0012] Figure 2 Another embodiment of a multi-chip module including a reconfigurable inference platform as described herein is illustrated;

[0013] Figure 3Another embodiment of a multi-chip module comprising a reconfigurable inference platform as described herein is illustrated;

[0014] Figure 4 A simplified functional block diagram of a reconfigurable inference platform as described herein is illustrated; and

[0015] Figure 5 A flowchart of a program executable by a reconfigurable inference platform as described herein.

[0016] Legend of the figures

[0017] 101, 201, 302, 401: processor chip

[0018] 102, 202, 301, 402: second memory chip

[0019] 103, 203, 303, 403: first memory chip

[0020] 110, 210, 310: adapter board

[0021] 111, 211, 311, 350: interconnect line

[0022] 112, 212, 240, 312, 352, 362: interface

[0023] 113, 122, 213, 222, 241, 313, 361: input / output interface

[0024] 120, 220, 320: multi-chip module

[0025] 121, 221: external contact structure

[0026] 131, 231, 332: processor-memory interface

[0027] 132, 232, 331: memory-processor interface

[0028] 410: central processing unit

[0029] 411: accelerator core

[0030] 412: on-chip memory

[0031] 413: first input / output interface

[0032] 414: second input / output interface

[0033] 420: bus system

[0034] 430: random access non-volatile memory

[0035] 431: memory input / output interface

[0036] 440: volatile memory

[0037] 441: first memory input / output interface

[0038] 450: vertical interconnect structure

[0039] 451: data path

[0040] 501, 502, 503, 504, 505, 506: steps DETAILED DESCRIPTION

[0041] The following references Figures 1-5 A detailed description of various embodiments of the present technology is provided.

[0042] Figure 1 A reconfigurable inference platform is illustrated, which includes a processor chip 101, a first memory chip 103 (model collection) and a second memory chip 102 (weight memory). In this example, the processor chip 101 and the second memory chip 102 are stacked, and the combination of the processor chip 101 and the second memory chip 102 stacked together is mounted on a board 110 with the first memory chip 103. The components are assembled as a multi-chip module 120 in a single package.

[0043] The processor chip 101 can include running processor cores (e.g., central processing units (CPUs)) and accelerator cores, such as artificial intelligence accelerators (e.g., AIAcc) or neuron processing units. The processor chip 101 includes a chip-to-chip bonding surface, on which a processor-memory interface 131 is exposed to connect the second memory chip 102. The second memory chip includes a memory-processor interface 132, which is exposed on a surface of the second memory chip and is complementary to the processor-memory interface 131 on the processor chip 101. In this example, a plurality of direct vertical connection structures are provided between the processor-memory interface and the memory-processor interface. The direct vertical connection structures can include copper via-to-via conductors or other suitable chip-to-chip contact technologies for high-speed, low-latency and low-power inter-chip communication.

[0044] In this example, the processor chip 101 includes an input / output interface 113 disposed on a surface of the processor chip 101. The input / output interface 113 is connected to a plurality of vertical connection structures, such as through silicon vias (TSVs), to connect the interconnect lines 111 on the board 110.

[0045] The first memory chip 103 includes an interface 112 to connect to the interconnect lines 111 on the interposer 110.

[0046] Thus, the interconnect lines 111 provide a portion of a data path between the first memory chip and the second memory chip, through the processor chip 101.

[0047] In Figure 1 In the illustrated example, the processor chip 101 includes another input / output interface 122 to connect to the external contact structures 121 of the multi-chip module 120.

[0048] Figure 2 Another configuration of an inference engine as described herein is illustrated. This configuration includes a processor chip 201, a first memory chip 203 (model collection) and a second memory chip 202 (weight memory). In this example, the processor chip 201 and the second memory chip 202 are stacked, and the combination of the processor chip 201 and the second memory chip 202 stacked together is mounted on an interposer 210 with the first memory chip 203. The assembly is a multi-chip module 220 in a single package.

[0049] The processor chip 201 can include a running processor core (e.g., a central processing unit) and an accelerator core, such as an artificial intelligence accelerator (e.g., an AI Acc) or a neuron processor. The processor chip 201 includes a chip-to-chip bonding surface, and a processor-to-memory interface 231 is exposed on the chip-to-chip bonding surface to connect to the second memory chip 202. The second memory chip includes a memory-to-processor interface 232 exposed on a surface of the second memory chip and complementary to the processor-to-memory interface 231 on the processor chip 201. In this example, a plurality of direct vertical connections are provided between the processor-to-memory interface and the memory-to-processor interface on the surfaces. The direct vertical connections can include very short copper via-to-via conductors or other chip-to-chip contact technology suitable for high-speed, low-latency, and low-power communication between chips.

[0050] In this example, the processor chip 201 includes an input / output interface 213 disposed on a surface of the processor chip 201. The input / output interface 213 is connected to vertical connections, such as through-silicon vias, to connect to the interconnect lines 211 on the interposer 210.

[0051] In addition, the second memory chip 202 includes an input / output interface 241 exposed on an opposite surface of the processor chip 201, which connects to a complementary interface 240 on the interposer 210 to connect to the interconnect lines 211 of the interposer 210.

[0052] The first memory chip 203 includes an interface 212 to connect to the interconnect lines 211 on the interposer 210.

[0053] Thus, the interconnect lines 211 provide a portion of a data path between the first memory chip and the second memory chip that is an alternative to the data path through the processor chip 201.

[0054] In this example, the processor chip 201 includes another input / output interface 222 to connect to the external contact structure 221 of the multi-chip module 220. Figure 2 In this example, the processor chip 302 includes an input / output interface 313 disposed on the lower surface of the processor chip 302. The input / output interface 313 is connected to vertical connections that connect to the interconnect lines 311 on the interposer 310.

[0055] Figure 3 Another configuration of an inference engine as described herein is illustrated. This configuration includes a processor chip 302, a first memory chip 303 (model collection) and a second memory chip 301 (weight memory). In this example, the processor chip 302 and the second memory chip 301 are stacked, and the combination of the processor chip 302 and the second memory chip 301 stacked together is mounted on an interposer 310 with the first memory chip 303. In this example, the processor chip 302 is between the second memory chip 301 and the interposer 310. The components are assembled as a multi-chip module 320 in a single package.

[0056] The processor chip 302 can include a running processor core (e.g., a central processing unit) and an accelerator core, such as an artificial intelligence accelerator (e.g., an AIAcc) or a neuron processor. The processor chip 302 includes a chip-to-chip bonding surface (upper surface) on which a processor-to-memory interface 332 is exposed to connect to the second memory chip 301. The second memory chip 301 includes a memory-to-processor interface 331 exposed on a surface (lower surface) of the second memory chip and complementary to the processor-to-memory interface 332 on the processor chip 302. In this example, a plurality of direct vertical connections are provided between the processor-to-memory interface and the memory-to-processor interface. The direct vertical connections can include copper via-to-via conductors or other suitable chip-to-chip contact technology for high speed, low latency and low power communication between chips.

[0057] In this example, the processor chip 302 includes an input / output interface 313 disposed on the lower surface of the processor chip 302. The input / output interface 313 is connected to vertical connections that connect to the interconnect lines 311 on the interposer 310.

[0058] In addition, the processor chip 302 includes an input / output interface 361 exposed on the lower surface opposite the second memory chip 301, which connects to a complementary interface 362 on the adapter board 310 to connect the interconnect lines 350 of the adapter board 310.

[0059] The first memory chip 303 includes an interface 312 to connect to the interconnect lines 311 on the adapter board 310.

[0060] Thus, the interconnect lines 311 provide a portion of the data path between the first memory chip and the second memory chip through the adapter board interconnect lines 311 and through the processor chip 302.

[0061] The adapter board 310 includes an interface 352 to connect to the interconnect lines 350 of the adapter board (which can connect to or be part of the interconnect lines 311 of the adapter board). Line connections are provided from the interface 352 to the external contact structures 351 of the multi-chip module 320.

[0062] In other embodiments, an interface or interfaces on a side surface or lower surface of the adapter board can be substituted for or in addition to the interface 352.

[0063] Figures 1-3 Exemplary configurations of the platform as described herein are provided, illustrating various configurations of packaged chips and connection structures in the chips, adapter boards, and external contacts. Other configurations can be implemented as desired.

[0064] Figure 4 To accompany Figures 1-3 A simplified functional block diagram of the platform is described. The platform includes a processor chip 401, a first memory chip 403, and a second memory chip 402. In this example, the processor chip 401 includes a central processor or processor core 410, an accelerator core 411, on-chip memory 412 (e.g., static random access memory (SRAM) that can be used as working memory and cache memory), a first input / output interface 413, and a second input / output interface 414. A bus system 420 provides intra-chip communications between the elements.

[0065] In this example, the first memory chip 403 includes high capacity non-volatile memory 440, such as three-dimensional NAND memory, for example implemented using charge-trapping storage technology. The first memory chip 403 includes a first memory input / output interface 441 for off-chip communication. Depending on the particular implementation of the memory chip used, the first memory input / output interface 441 can include a high-speed serial port, such as a serial peripheral interface compatible port (SPI compatible port), or a parallel port. In this example, a data path 451 is provided between the first memory input / output interface 441 and the first input / output interface 413 on the processor chip 401.

[0066] In this example, the second memory chip 402 includes high-speed random access non-volatile memory 430, such as three-dimensional phase change storage technology. In other examples, the second memory chip 402 can include NOR flash memory, or other suitable random access technology, such as resistive random access memory (e.g., metal-oxide memory), magnetoresistive random access memory, ferroelectric random access memory, etc., using charge-trapping storage technology.

[0067] The second memory chip 402 includes a memory input / output interface 431 for off-chip communication, which is connected directly to the second input / output interface 414 on the processor chip 401 through a plurality of vertical interconnect structures 450.

[0068] When on-chip static random access memory is not large enough, dynamic random access memory (DRAM) can be selected to be integrated into the system-in-a-package. Thermal management can be used to ensure the preservation of data.

[0069] As used herein, an accelerator core (e.g., accelerator core 411) is a configurable logic circuit that includes a plurality of elements designed to perform or are adapted to perform arithmetic operations for a portion or all of an inference model. The configuration of the accelerator core can include loading a set of weights or a portion of a set of weights for the inference model. In some embodiments, the configuration of the accelerator core can include loading a portion or all of the computation graphs of the inference model to define the order of operations and the architecture of the computation graphs of the inference model. The inference model can include a computation graph of a deep learning neural network that, in some examples, has a plurality of fully connected and partially connected layers, activation functions, normalization functions, and the like.

[0070] The accelerator core can be implemented using configurable logic, such as an array of configurable cells for field programmable gate arrays, where a compiled computation graph is configured in bit files. The accelerator core can use a hybrid of data flow configurable logic and sequential processing configurable logic.

[0071] A runtime processor core (e.g., central processor 410) can execute a runtime program to coordinate the operations of the accelerator core to achieve an inference operation, including data input / output operations, loading computation graphs, moving a set of weights for an inference operation into and out of the accelerator core, transferring input data to the accelerator core, and performing a portion of the computation graph.

[0072] Figure 5 An example flowchart of the logic of a runtime program for execution by a reconfigurable inference platform, such as in conjunction with Figures 1-4The platform logic can be implemented using computer programs stored in memory, such as on-chip memory 412 or other memory accessed by central processor 410. In this example, the program includes downloading a set of executable artificial intelligence models from an external source, such as a network, and loading the set of executable artificial intelligence models in high-capacity NAND flash memory on the platform (step 501). During operation, the program waits for a control event (step 502). Control events can include a reset, expiration of a timer, receipt of information from a communication network or other external source, data generated by an inference engine executing in the processor chip itself, or other signals. As long as no control event is measured, the program loops.

[0073] When a control event is measured, the program includes selecting an artificial intelligence model from the set of executable artificial intelligence models stored in NAND flash memory (step 503). Then, the selected model or at least a set of weights of the selected model is transferred from the NAND flash memory to the weight memory (step 504). The program includes configuring the accelerator core using parameters of the selected model read from the NAND flash memory (step 505). After loading the weights and configuring the accelerator core, the program includes executing an inference program using parameters of the selected model stored in the weight memory, including transferring parameters (such as weights) between processor chip 401 and second memory chip 402 using multiple direct vertical connections (step 506).

[0074] Thus, Figure 5 The program includes selecting an executable model from the set of executable models stored in the first memory chip to load a computation graph for the selected model (including configuring the accelerator core), to cause a set of weights of the selected model to be transferred to the second memory chip, and to execute the selected model. In addition, as Figure 5 indicated, after executing or beginning execution of the selected model, the program loops to step 502 to wait for a next control event. Upon measuring the next control event, steps 503-506 are performed in sequence, and can include changing the selected model to a different model in the set of executable models, loading a computation graph for the different model (including configuring the accelerator core), causing a set of weights of the different model to be transferred to the second memory chip, and executing the different model.

[0075] In conjunction Figure 5It will be appreciated that many of the steps can be combined, done simultaneously, or in a different order without affecting the efficacy achieved. In some cases, as the reader will appreciate, reordering of multiple steps can achieve the same result only if certain other changes occur. Further, it will be appreciated that the flowcharts herein illustrate only steps relevant to the present application, but that many additional steps can be performed before, after, and in between the illustrated steps to achieve other efficacy.

[0076] The present application describes a system-in-a-package platform in which one or more three-dimensional NAND chip stores a collection containing multiple distinct artificial intelligence models (computation graphs and weights), a weight memory chip stores weights of a selected artificial intelligence model, and a processor chip, which can be a special purpose artificial intelligence logic chip (central processing unit and artificial intelligence accelerator), is included in the memory system to execute selected artificial intelligence model parameters (e.g., weights), hyperparameters (e.g., neural network computation graph or architecture details) required by the central processing unit / network processor (e.g., layers, normalization functions, activation functions, etc.).

[0077] Inter-chip bonding between the artificial intelligence logic chip and the weight memory chip can employ via-to-via copper bonding or other 3D (2.5D) bonding techniques.

[0078] While the application has been disclosed by reference to the preferred embodiments and examples described above, it should be understood that these are presented by way of example only, and that numerous changes can be made to the application without departing from the spirit thereof as recited in the appended claims.

Claims

1. A reconfigurable inference system, comprising: An apparatus comprising: a processor chip including a running processor core, an accelerator core, on-chip memory, and a processor-memory interface exposed on a chip-to-chip bonding surface of the processor chip; a first memory chip accessible by the processor chip to store a set of executable models of an inference engine, each executable model of the set of executable models including a set of weights for executing the executable model; a second memory chip storing the set of weights of a selected executable model, the second memory chip including a memory-processor interface exposed on a surface of the second memory chip and complementary to the processor-memory interface; and a plurality of direct vertical connection structures between the processor-memory interface and the memory-processor interface; wherein the running processor core accesses an instruction memory storing executable instructions to perform a program including selecting an executable model from the set of executable models stored in the first memory chip, loading a computation graph for the selected executable model, causing the set of weights of the selected executable model to be transferred to the second memory chip, and executing the selected executable model, wherein loading the computation graph for the selected executable model includes configuring the accelerator core. The direct vertical connection structures include a plurality of via-to-via connection structures.

2. The reconfigurable inference system of claim 1, wherein, The running processor core accesses an instruction memory storing executable instructions to perform a program in response to a control event, the program including:

3. The reconfigurable inference system of claim 1, wherein, causing the selected executable model to change to a different executable model of the set of executable models, loading a computation graph for the different executable model, causing the set of weights of the different executable model to be transferred to the second memory chip, and executing the different executable model, wherein loading the computation graph for the different executable model includes configuring the accelerator core. The processor chip and the second memory chip are stacked and disposed on an interposer, the first memory chip is disposed on the interposer, the interposer includes an interconnect that forms a portion of a data path between the first memory chip and the second memory chip.

4. The reconfigurable inference system of claim 1, wherein, The processor chip includes a second input / output interface, the data path includes a connection from the interconnect of the interposer to the second input / output interface on the processor chip.

5. The reconfigurable inference system of claim 4, wherein, The interposer is located below the second memory chip, and the processor chip is disposed above the second memory chip.

6. The reconfigurable inference system of claim 4, wherein, The interposer is located below the processor chip, and the second memory chip is disposed above the processor chip.

7. The reconfigurable inference system of claim 4, wherein, The first memory chip includes a charge-trapping NAND architecture memory, and the second memory chip includes a non-volatile random access memory.

8. The reconfigurable inference system of claim 1, wherein, The non-volatile random access memory is a phase change memory.

9. The reconfigurable inference system of claim 8, wherein, ​

Images