Memory chips connecting system-on-chips and accelerator chips

Abstract

The memory chip connects the SoC and the accelerator chip. The system includes the memory chip and the accelerator chip. The system may include an SoC. The memory chip may include a first memory cell that stores and provides computational input data received from the SoC, where the computational input data can be used by the accelerator chip as computational input. The memory chip may include a second memory cell that stores and provides first computational output data received from the accelerator chip, where the first computational output data can be obtained by the SoC or reused by the accelerator chip as computational input. The memory chip may also include a third memory cell that stores second computational output data related to a non-AI task received from the SoC, where the second computational output data can be obtained by the SoC for the non-AI task.

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. patent application Ser. No. 16 / 573,805, filed Sep. 17, 2019, and entitled “MEMORY CHIP CONNECTING A SYSTEM ON A CHIP AND AN ACCELERATOR CHIP,” the entire disclosure of which is incorporated herein by reference.

[0002] At least some embodiments disclosed herein relate to a memory chip connecting an SoC and an accelerator chip (e.g., an AI accelerator chip). At least some embodiments disclosed herein relate to forming a memory using a memory hierarchy and memory chip strings. [Background technology]

[0003] Memory, such as main memory, is computer hardware that stores information for immediate use in a computer or computing device. Memory generally operates faster than computer storage, which allows for slower access to information but also allows for greater capacity and more reliable data. One type of memory, random access memory (RAM), can have high operating speeds.

[0004] Typically, memories are composed of addressable semiconductor memory units or cells. Memory ICs and their memory units can be implemented at least in part by silicon-based metal-oxide-semiconductor field-effect transistors (MOSFETs).

[0005] There are two main types of memory: volatile and non-volatile. Non-volatile memory can include flash memory (which can also be used as storage), as well as ROM, PROM, EPROM, and EEPROM (which can be used to store firmware). Another type of non-volatile memory is non-volatile random access memory (NVRAM). Volatile memory can include main memory technologies such as dynamic random access memory (DRAM), and cache memory, which is usually implemented using static random access memory (SRAM).

[0006] An AI accelerator is a type of microprocessor or computer system that is configured to speed up calculations for AI applications, including AI applications such as artificial neural networks, machine vision, and machine learning. AI accelerators can be hardwired to improve data processing for data-intensive or sensor-driven tasks. AI accelerators can include one or more cores and can be hardwired for low-precision arithmetic and in-memory computing. AI accelerators can be found in many devices, such as smartphones, tablets, and computers of all types, especially those with sensors and data-intensive tasks such as graphics and optical processing. AI accelerators can also include vector processors or array processors to improve performance for numerical simulations and other types of tasks used in AI applications.

[0007] An SoC is an integrated circuit (IC) that integrates computer components onto a single chip. Typical computer components in an SoC include a central processing unit (CPU), memory, input / output ports, and secondary storage. An SoC can contain all of its components on a single substrate or microchip, with some chips being smaller than a quarter. An SoC can have a variety of signal processing capabilities and can include dedicated processors or co-processors, such as a graphics processing unit (GPU). The tight integration allows an SoC to consume much less power than a traditional multi-chip system with comparable functionality. This makes an SoC beneficial for the integration of mobile computing devices (such as in smartphones and tablets). An SoC can also be useful for embedded systems and the Internet of Things, especially when smart devices are small.

[0008] Returning to memory, the memory of a computing system can be a hierarchy. Often referred to as a memory hierarchy in computer architecture, a memory hierarchy can separate computer memory into hierarchies based on certain factors such as response time, complexity, capacity, persistence, and memory bandwidth. Factors such as these can be related and often trade-offs that further emphasize the usefulness of a memory hierarchy.

[0009] In general, memory hierarchy affects performance in a computer system. Prioritizing memory bandwidth and speed over other factors may require consideration of memory hierarchy limitations such as response time, complexity, capacity, and persistence. To manage such prioritization, different types of memory chips may be combined to balance faster chips with more reliable or cost-effective chips, etc. Each of the different chips may be considered part of a memory hierarchy. And, for example, to reduce latency in a faster chip, other chips in the memory chip combination may respond by filling buffers and then sending signals to activate data transfer between the chips.

[0010] The memory hierarchy can be composed of chips that contain various types of memory units or cells. For example, a memory cell can be a DRAM unit. DRAM is a type of random access semiconductor memory that stores each bit of data in a memory cell, which usually includes a capacitor and a MOSFET. The capacitor can either be charged or discharged, which is represented by two values ​​in a bit, such as "0" and "1". In DRAM, the charge on the capacitor leaks away, so DRAM requires an external memory refresh circuit that periodically rewrites the data in the capacitor by restoring the original charge to each capacitor. DRAM is considered a volatile memory because its data is quickly lost when power is removed. This differs from other types of non-volatile memory, such as flash memory and NVRAM, where data storage is more permanent.

[0011] One type of NVRAM is 3D XPoint memory, in which memory units store bits based on changes in bulk resistance in combination with a stackable cross-lattice data access array. 3D XPoint memory can be more cost-effective than DRAM, but less cost-effective than flash memory. 3D XPoint is also a non-volatile memory and a random access memory.

[0012] Flash memory is another type of non-volatile memory. The advantage of flash memory is that it can be electrically erased and reprogrammed. Flash memory is considered to have two main types, NAND flash memory and NOR flash memory, named after the NAND and NOR logic gates that can implement the memory units of flash memory. Flash memory units or cells exhibit internal characteristics similar to those of the corresponding gates. NAND flash memory contains NAND gates. NOR flash memory contains NOR gates. NAND flash memory may be written to and read from in blocks that can be smaller than the entire device. NOR flash allows single bytes to be written to erased locations or read independently. The advantages of NAND flash memory make such memories popular in memory cards, USB flash drives, and solid-state drives. However, the main tradeoff of using flash memory in general is that a given block can only have a relatively small number of write cycles compared to other types of memory such as DRAM and NVRAM.

[0013] The present disclosure will become more fully understood from the following detailed description and the accompanying drawings of various embodiments of the present disclosure. [Brief description of the drawings]

[0014] [Figure 1]1 illustrates an example of an associated system that includes an accelerator chip (e.g., an AI accelerator chip) that connects an SoC and a memory chip. [Diagram 2] 2 shows an example of an associated system including the accelerator chip shown in FIG. 1 and a separate memory. [Diagram 3] 2 shows an example of an associated system including the accelerator chip shown in FIG. 1 and a separate memory. [Figure 4] 1 illustrates an example of a system according to some embodiments of the present disclosure that includes a memory chip connecting an SoC and an accelerator chip (e.g., an AI accelerator chip). [Diagram 5] 5 illustrates an example of a system including the memory chip shown in FIG. 4 and a separate memory. [Figure 6] 5 illustrates an example of a system including the memory chip shown in FIG. 4 and a separate memory. [Figure 7] 5 illustrates an example of a system including the memory chip shown in FIG. 4 and a separate memory. [Figure 8] 1 illustrates an example of a parts arrangement of an exemplary computing device, according to some embodiments of the present disclosure. [Figure 9] 1 illustrates another example of a parts arrangement for an exemplary computing device, according to some embodiments of the present disclosure. [Figure 10] 1 shows an example of a memory chip string that can be used for the separate memories shown in FIGS. [Figure 11] 1 shows an example of a memory chip string that can be used for the separate memories shown in FIGS. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] At least some embodiments disclosed herein relate to a memory chip (e.g., DRAM) that interfaces with an SoC and an accelerator chip (e.g., an AI accelerator chip). In other words,At least some embodiments disclosed herein relate to connecting an accelerator chip (e.g., an AI accelerator chip) to a SoC via a memory chip. The accelerator chip communicates with the SoC indirectly via the memory chip. Data placed by the memory chip on the memory chip connecting the SoC and the accelerator chip is interpreted as a request to the accelerator chip. The SoC may also optionally use the memory chip connecting the SoC and the accelerator chip for its operations that do not involve the accelerator chip. Thus, the memory chip connecting the SoC and the accelerator chip may have two general purposes: one for use with the SoC and one for use with the accelerator chip. For some examples of such embodiments, see first memory chip 402, accelerator chip 404, and SoC 406 shown in FIGS. 4-7. Also, see SoC 806 and application-specific components 807 shown in FIGS. 8-9. The application specific components 807 may include, in some embodiments of the devices 800 and 900, the first memory chip 402 and the accelerator chip 404.

[0016] As shown in FIGS. 4-7, the memory chip connecting the SoC and the accelerator chip can be logically (and sometimes physically) between the SoC and the accelerator chip. Also, the memory chip for the accelerator that is between the SoC and the accelerator chip may not need to have two sets of pins. In some embodiments, the accelerator chip and the memory chip can be physically on the same bus. However, in any situation where an intermediate memory chip is used, the SoC does not directly communicate with the accelerator chip via the bus or wiring. Thus, the memory chip connecting the SoC and the accelerator chip is at least logically between the accelerator chip and the SoC. Also, the connection between the SoC and the accelerator chip provided by the memory chip may only be a logical connection.

[0017] The memory chip connecting the SoC and the accelerator chip may include two separate sets of pins: one set for connecting directly to the accelerator chip via wiring (see, e.g., set of pins 414 and wiring 424 shown in FIGS. 4, 5, and 7), and another set for connecting directly to the SoC via wiring (see, e.g., set of pins 416 and wiring 426 shown in FIGS. 4-5).

[0018] When the accelerator chip is connected to the SoC through the memory chip, application-specific computations (such as AI computations) for the SoC can be accelerated, generally, or more specifically, in some embodiments, for a GPU (see, e.g., GPU 408 shown in FIGS. 4-7) included in the SoC. In some embodiments, the GPU in the SoC can be directly connected to the memory chip connecting the SoC and the accelerator chip. In some embodiments, the memory chip connecting the GPU and the accelerator chip can include a set of pins and can be directly connected to the accelerator chip through the set of pins and wiring (see, e.g., set of pins 414 and wiring 424). The accelerator chip can also include a corresponding set of pins (see, e.g., set of pins 415). And, the memory chip connecting the SoC and the accelerator chip can include a second set of pins and can be directly connected to the GPU through the second set of pins and wiring (see, e.g., set of pins 416 and wiring 426). Also, the GPU in the SoC may include a set of pins and may be directly connected to the memory chips via this set of pins and wires (see, for example, set of pins 417 and wires 426).

[0019] For purposes of this disclosure, any one of the accelerator chips described herein is a dedicated accelerator chip. in To be, or Containing or being part ofIt should be understood that the dedicated accelerator chips may include an artificial intelligence (AI) accelerator chip, a virtual reality accelerator chip, an augmented reality accelerator chip, a graphics accelerator chip, a machine learning accelerator chip, or any other type of ASIC or FPGA that may provide low latency or high bandwidth memory access. For example, any one of the accelerator chips described herein may be an AI accelerator chip. in To be, or Containing or being part of It is possible.

[0020] The accelerator chip can be a microprocessor chip or SoC itself designed for hardware acceleration of AI applications, including artificial neural networks, machine vision, and machine learning. In some embodiments, the accelerator chip is configured to perform numerical calculations on vectors and matrices (e.g., see vector processor 412 shown in FIG. 4, which can be configured to perform numerical calculations on vectors and matrices). The accelerator chip can be or include an ASIC or FPGA. In ASIC embodiments of the accelerator chip, the accelerator chip can be hardwired dedicated to accelerating application-specific calculations (such as AI calculations). In some other embodiments, the accelerator chip can be a modified FPGA or GPU that has been modified for acceleration of application-specific calculations beyond an unmodified FPGA or GPU. In some other embodiments, the accelerator chip can be an unmodified FPGA or GPU.

[0021] The memory chips directly connected to the accelerator chip (e.g., see the first memory chip 402) are also referred to herein as application-specific memory chips for clarity in describing the multiple memory chips in the entire system. The application-specific memory chips are not necessarily hardwired for dedicated application-specific computation (such as AI computation). Each of the application-specific memory chips can be a DRAM chip or an NVRAM chip. And each of the application-specific memory chips can be directly connected to the accelerator chip and can include memory units dedicated to accelerating the application-specific computation by the accelerator after the application-specific memory chips are configured by the SoC or the accelerator chip.

[0022] In some embodiments, the SoC may include a main processor (e.g., a CPU). See, for example, main processor 110 shown in FIGS. 4-7. In such embodiments, the GPU in the SoC may execute instructions for application-specific tasks and calculations (e.g., AI tasks and calculations), and the main processor may execute instructions for non-application-specific tasks and calculations (e.g., non-AI tasks and calculations). And, in such embodiments, the accelerator may provide acceleration of application-specific tasks and calculations dedicated to the GPU. The SoC may also include its own bus for interconnecting components of the SoC (e.g., connecting the main processor and the GPU). Moreover, the bus of the SoC may be configured to connect the SoC to a bus external to the SoC, so that components of the SoC may be coupled with chips and devices external to the SoC, such as separate memory chips.

[0023] Non-application-specific computations and tasks of the GPU (e.g., non-AI computations and tasks), or such computations and tasks that do not use an accelerator chip, may not be traditional tasks performed by the main processor, but may use separate memory (which may be application-specific memory), such as a separate memory chip. And the memory may be implemented with DRAM, NVRAM, flash memory, or any combination thereof. For example, the separate memory or memory chip may be connected to the SoC and the main processor via a bus external to the SoC (see, for example, memory 204 and bus 202 shown in FIG. 5). In such embodiments, the separate memory or memory chip may have a memory unit dedicated to the main processor. Also, the separate memory or memory chip may be connected to the SoC and the GPU via a bus external to the SoC (see, for example, second memory chip 204 and bus 202 shown in FIGS. 5-7). In such embodiments, the separate memory or memory chip may include a memory unit in the main processor or the GPU.

[0024] For purposes of this disclosure, it should be understood that the application specific memory chips and the separate memory chips can each be replaced by a group of memory chips, such as a memory chip string (see, for example, the memory chip strings shown in FIGS. 10 and 11). For example, the separate memory chips can be replaced by a memory chip string including at least an NVRAM chip and a flash memory chip downstream of the NVRAM chip. Also, the separate memory chips can be replaced by at least two memory chips, one of which is for the main processor (e.g., CPU) and another chip is for the GPU to use as memory for non-AI computations and / or tasks.

[0025] Additionally, at least some embodiments disclosed herein relate to accelerator chips (e.g., AI accelerator chips) having vector processors (see, e.g., vector processor 412 shown in FIGS. 4-7), and at least some embodiments disclosed herein relate to forming memories using memory hierarchies and memory chip strings (see, e.g., FIGS. 10 and 11).

[0026] For purposes of this disclosure, any one of the accelerator chips described herein is a dedicated accelerator chip. in To be, or Containing or being part of It should be understood that examples of dedicated accelerator chips can include an AI accelerator chip, a virtual reality accelerator chip, an augmented reality accelerator chip, a graphics accelerator chip, a machine learning accelerator chip, or any other type of ASIC or FPGA that can provide low latency or high bandwidth memory access.

[0027] FIG. 1 illustrates an example of an associated system that includes an accelerator chip (e.g., an AI accelerator chip) that connects an SoC and memory chips.

[0028] 1 illustrates an exemplary system 100, which is related to system 400 in part. System 100 includes an accelerator chip 102 (e.g., an AI accelerator chip) that couples a first memory chip 104 and a SoC 106. As shown, SoC 106 includes a GPU 108 and a main processor 110. Main processor 110 may be or include a CPU. And accelerator chip 102 includes a vector processor 112.

[0029] In system 100, accelerator chip 102 includes a first set of pins 114 and a second set of pins 116. The first set of pins 114 is configured to connect to a first memory chip 104 via interconnects 124. The second set of pins 116 is configured to connect to a SoC 106 via interconnects 126. As shown, first memory chip 104 includes a corresponding set of pins 115 that connects the memory chip to accelerator chip 102 via interconnects 124. GPU 108 of SoC 106 includes a corresponding set of pins 117 that connects the SoC to accelerator chip 102 via interconnects 126.

[0030] The accelerator chip 102 is configured to execute and accelerate application-specific computations (e.g., AI computations) of the SoC 106. The accelerator chip 102 is also configured to use the first memory chip 104 as a memory for the application-specific computations. The acceleration of the application-specific computations can be performed by the vector processor 112. The vector processor 112 in the accelerator chip 102 can be configured to execute numerical computations on vectors and matrices for the SoC 106. The accelerator chip 102 can include an ASIC that includes the vector processor 112 and is dedicated hardwired to accelerate the application-specific computations (e.g., AI computations) via the vector processor 112. Alternatively, the accelerator chip 102 can include an FPGA that includes the vector processor 112 and is dedicated hardwired to accelerate the application-specific computations via the vector processor 112. In some embodiments, the accelerator chip 102 can include a GPU that includes the vector processor 112 and is dedicated hardwired for accelerating application-specific computations via the vector processor 112. In such embodiments, the GPU can be dedicated and modified for accelerating application-specific computations via the vector processor 112.

[0031] As shown, the SoC 106 includes a GPU 108. And, the accelerator chip 102 can be configured to perform and accelerate application-specific calculations (e.g., AI calculations) for the GPU 108. For example, the vector processor 112 can be configured to perform numerical calculations on vectors and matrices for the GPU 108. And, the GPU 108 can be configured to perform application-specific tasks and calculations (e.g., AI tasks and calculations).

[0032] As also shown, SoC 106 includes a main processor 110 that is configured to perform non-AI tasks and calculations.

[0033] In some embodiments, the memory chip 104 is a DRAM chip. In such examples, the first set of pins 114 can be configured to connect to the DRAM chip via the wiring 124. The accelerator chip 102 can also be configured to use DRAM cells in the DRAM chip as memory for application-specific computations (e.g., AI computations). In some other embodiments, the memory chip 104 is an NVRAM chip. In such examples, the first set of pins 114 can be configured to connect to the NVRAM chip via the wiring 124. The accelerator chip 102 can also be configured to use NVRAM cells in the NVRAM chip as memory for application-specific computations. Furthermore, the NVRAM chip can be or include a 3D XPoint memory chip. In such examples, the first set of pins 114 can be configured to connect to the 3D XPoint memory chip via the wiring 124, and the accelerator chip 102 can be configured to use 3D XPoint memory cells in the 3D XPoint memory chip as memory for application-specific computations.

[0034] In some embodiments, the system 100 includes an accelerator chip 102 connected via wires to a first memory chip 104, which can be an application-specific memory chip. The system 100 also includes a SoC 106, which includes a GPU 108 (which can be configured to perform AI tasks) and a main processor 110 (which can be configured to perform non-AI tasks and delegate AI tasks to the GPU 108). In such embodiments, the GPU 108 includes a set of pins 117 configured to connect via wires 126 to the accelerator chip 102, which is configured to perform and accelerate AI computations of the AI ​​tasks for the GPU 108.

[0035] In such embodiments, the accelerator chip 102 may include a vector processor 112 configured to perform numerical calculations on vectors and matrices for the GPU 108. And the accelerator chip 102 may include an ASIC that includes the vector processor 112 and is dedicated hardwired to accelerate AI calculations via the vector processor 112. Or the accelerator chip 102 may include an FPGA that includes the vector processor 112 and is dedicated hardwired to accelerate AI calculations via the vector processor 112. Or the accelerator chip 102 may include a GPU that includes the vector processor 112 and is dedicated hardwired to accelerate AI calculations via the vector processor 112.

[0036] The system 100 also includes a memory chip 104, and the accelerator chip 102 can be connected to the memory chip 104 via wiring 124 and can be configured to perform and accelerate AI computations of AI tasks. The memory chip 104 can be or include a DRAM chip having DRAM cells, and the DRAM cells can be configured by the accelerator chip 102 to store data for accelerating the AI ​​computations. Or, the memory chip 104 can be or include a NVRAM chip having NVRAM cells, and the NVRAM cells can be configured by the accelerator chip 102 to store data for accelerating the AI ​​computations. The NVRAM chip can include 3D XPoint memory cells, and the 3D XPoint memory cells can be configured by the accelerator chip 102 to store data for accelerating the AI ​​computations.

[0037] 2-3 show exemplary systems 200 and 300, respectively, and examples of separate memory (eg, NVRAM), each system including the accelerator chip 102 shown in FIG.

[0038] 2, bus 202 connects system 100 (including accelerator chip 102) to memory 204, which in some embodiments can be NVRAM and is separate memory from the memory of first memory chip 104 of system 100, and which in some embodiments can be main memory.

[0039] In the system 200, the SoC 106 of the system 100 is connected to the memory 204 via a bus 202. The system 100 as a part of the system 200 includes the accelerator chip 102, the first memory chip 104, and the SoC 106. These parts of the system 100 are connected to the memory 204 via the bus 202. As shown in FIG. 2, the memory controller 206 included in the SoC 106 controls data access to the memory 204 by the SoC 106 of the system 100. For example, the memory controller 206 controls data access to the memory 204 by the GPU 108 and / or the main processor 110. In some embodiments, the memory controller 206 can control data access to all memories in the system 200 (such as data access to the first memory chip 104 and the memory 204). The memory controller 206 can be communicatively coupled to the first memory chip 104 and / or the memory 204.

[0040] The memory 204 is separate from the memory provided by the first memory chip 104 of the system 100, and it can be used as memory by the GPU 108 and main processor 110 of the SoC 106 via the memory controller 206 and the bus 202. The memory 204 can also be used by the GPU 108 and the main processor 110 as memory for non-application specific or application specific tasks (such as non-AI or AI tasks) that are not executed by the accelerator chip 102. Data for such tasks can be accessed by and communicated to and from the memory 204 via the memory controller 206 and the bus 202.

[0041] In some embodiments, memory 204 is a main memory of a device, such as a device that hosts system 200. For example, in system 200, memory 204 can be main memory 808 shown in FIG.

[0042] 3, a bus 202 connects the system 100 (including the accelerator chip 102) to the memory 204. And in system 300, the bus 202 connects the accelerator chip 102 to the SoC 106 and connects the accelerator chip 102 to the memory 204. Also shown, in system 300, the bus 202 replaces the second set of pins 116 of the accelerator chip, as well as the wiring 126 and the set of pins 117 of the SoC 106 and the GPU 108. The accelerator chip 102 in system 300 connects the first memory chip 104 and the SoC 106 of system 100, similar to system 200, except that this connection is via the first set of pins 114 and the bus 202.

[0043] Also, similar to the system 200, in the system 300, the memory 204 is a separate memory from the memory of the first memory chip 104 of the system 100. In the system 300, the SoC 106 of the system 100 is connected to the memory 204 via the bus 202. And, in the system 300, the system 100 as a part of the system 300 includes the accelerator chip 102, the first memory chip 104, and the SoC 106. These parts of the system 100 are connected to the memory 204 via the bus 202 in the system 300. Also, similar to that shown in FIG. 3, the memory controller 206 included in the SoC 106 controls data access of the memory 204 by the SoC 106 of the system 100. In some embodiments, the memory controller 206 can control data access of all memories in the system 300 (such as data access of the first memory chip 104 and the memory 204). And, the memory controller can be connected to the first memory chip 104 and / or the memory 204. And, the memory controller 206 can be communicatively coupled to the first memory chip 104 and / or the memory 204 .

[0044] Also, in system 300, memory 204 (which may be NVRAM in some embodiments) is separate from the memory provided by the first memory chip 104 of system 100, and it may be used as memory by the GPU 108 and main processor 110 of SoC 106 via memory controller 206 and bus 202. Additionally, accelerator chip 102 may use memory 204 via bus 202 in some embodiments and situations. Memory 204 may then be used by GPU 108 and main processor 110 as memory for non-application specific or application specific tasks (such as non-AI or AI tasks) not executed by accelerator chip 102. Data for such tasks may be accessed by and communicated to and from memory 204 via memory controller 206 and / or bus 202.

[0045] In some embodiments, the memory 204 is a main memory of a device, such as the device that hosts the system 300. For example, in the system 300, the memory 204 can be the main memory 808 shown in FIG.

[0046] 4 illustrates an example of a system 400 including a first memory chip 402 connecting an accelerator chip 404 (e.g., an AI accelerator chip) and a SoC 406, in accordance with some embodiments of the present disclosure. As shown, the SoC 406 includes a GPU 408 and a main processor 110. The main processor 110 may be or include a CPU within the system 400. In turn, the accelerator chip 404 includes a vector processor 412.

[0047] In system 400, memory chip 402 includes a first set of pins 414 and a second set of pins 416. The first set of pins 414 is configured to connect to accelerator chip 404 via lines 424. The second set of pins 416 is configured to connect to SoC 406 via lines 426. As shown, accelerator chip 404 includes a corresponding set of pins 415 that connects the first memory chip 402 to the accelerator chip via lines 424. GPU 408 of SoC 406 includes a corresponding set of pins 417 that connects the SoC to first memory chip 402 via lines 426.

[0048] The first memory chip 402 includes a plurality of first memory cells configured to store and provide computational input data (e.g., AI computational input data) received from the SoC 406 via the second set of pins 416 for use by the accelerator chip 404 as computational input (e.g., AI computational input). The computational input data is accessed from the plurality of first memory cells and transmitted from the first memory chip 402 via the first set of pins 414 for receipt and use by the accelerator chip 404. The plurality of first memory cells may include DRAM cells and / or NVRAM cells. In an example having NVRAM cells, the NVRAM cells are or may include 3D XPoint memory cells.

[0049] The first memory chip 402 also includes a plurality of second memory cells configured to store and provide computation output data (e.g., AI computation output data) received from the accelerator chip 404 via the first set of pins 414, which is retrieved by the SoC 406 or reused by the accelerator chip 404 as computation input (e.g., AI computation input). The computation output data can be accessed from the plurality of second memory cells, transmitted from the first memory chip 402 via the first set of pins 414, and received and used by the accelerator chip 404. The computation output data can also be accessed from the plurality of second memory cells, transmitted from the SoC 406 or a GPU 408 within the SoC via the second set of pins 416, and received and used by the SoC or a GPU within the SoC. The plurality of second memory cells can include DRAM cells and / or NVRAM cells. In an example having NVRAM cells, the NVRAM cells are or can include 3D XPoint memory cells.

[0050] The first memory chip 402 also includes a plurality of third memory cells configured to store non-AI data associated with the non-AI tasks received from the SoC 406 via a set of pins 416, which is retrieved by the SoC 406 for the non-AI tasks. The non-AI data can be accessed from the plurality of third memory cells and transmitted from the first memory chip 402 via a second set of pins 416 and received and used by the SoC 406, the GPU 408 in the SoC, or the main processor 110 in the SoC. The plurality of third memory cells can include DRAM cells and / or NVRAM cells. In an example having NVRAM cells, the NVRAM cells are or can include 3D XPoint memory cells.

[0051] The accelerator chip 404 is configured to perform and accelerate application-specific computations (e.g., AI computations) for the SoC 406. The accelerator chip 404 is also configured to use the first memory chip 402 as a memory for the application-specific computations. The acceleration of the application-specific computations can be performed by a vector processor 412. The vector processor 412 in the accelerator chip 404 can be configured to perform numerical computations on vectors and matrices for the SoC 406. For example, the vector processor 412 can be configured to perform numerical computations on vectors and matrices for the SoC 406 using a plurality of first memory cells and a plurality of second memory cells as memory.

[0052] The accelerator chip 404 may include an ASIC that includes a vector processor 412 and is dedicated hardwired to accelerate application-specific computations (e.g., AI computations) via the vector processor 412. Alternatively, the accelerator chip 404 may include an FPGA that includes a vector processor 412 and is dedicated hardwired to accelerate application-specific computations via the vector processor 412. In some embodiments, the accelerator chip 404 may include a GPU that includes a vector processor 412 and is dedicated hardwired to accelerate application-specific computations via the vector processor 412. In those such embodiments, the GPU may be dedicatedly modified to accelerate application-specific computations via the vector processor 412.

[0053] As shown, the SoC 406 includes a GPU 408, and the accelerator chip 402 can be configured to perform and accelerate application-specific calculations for the GPU 408. For example, the vector processor 412 can be configured to perform numerical calculations on vectors and matrices for the GPU 408, and the GPU 408 can be configured to perform application-specific tasks and calculations. Also as shown, the SoC 406 includes a main processor 110, which is configured to perform non-AI tasks and calculations.

[0054] In some embodiments, the system 400 includes a memory chip 402, an accelerator chip 404, and a SoC 406, where the memory chip 402 includes at least a first set of pins 414 configured to connect to the accelerator chip 404 via lines 424 and a second set of pins 416 configured to connect to the SoC 406 via lines 426. The memory chip 402 may include a first plurality of memory cells configured to store and provide AI computation input data received from the SoC 406 via one set of pins 416 for use by the accelerator chip 404 as AI computation inputs, and a second plurality of memory cells configured to store and provide AI computation output data received from the accelerator chip 404 via another set of pins 414 for capture by the SoC 406 or reused by the accelerator chip 404 as AI computation inputs. The memory chip 402 may include a third plurality of cells used for memory for non-AI computations.

[0055] The SoC 406 also includes a GPU 408, and the accelerator chip 404 can be configured to perform and accelerate AI calculations for the GPU 408 using the first and second memory cells as memory. The accelerator chip 404 also includes a vector processor 412, and the vector processor can be configured to perform numerical calculations on vectors and matrices for the SoC 406 using the first and second memory cells as memory.

[0056] Also in system 400, a first number of memory cells in memory chip 402 are configured to store and provide AI computation input data received from SoC 406 via one set of pins 416, which can be used by accelerator chip 404 (e.g., an AI accelerator chip) as AI computation input. And a second number of memory cells in memory chip 402 are configured to store and provide AI computation output data received from accelerator chip 404 via another set of pins 414, which can be retrieved by SoC 406 or reused by accelerator chip 404 as AI computation input. And a third number of memory cells in memory chip 402 are configured to store non-AI data related to non-AI tasks received from SoC 406 via one set of pins 416, which can be retrieved by SoC 406 for the non-AI tasks.

[0057] The first memory cells, the second memory cells, and the third memory cells in the memory chip 402 may each include DRAM cells and / or NVRAM cells, and the NVRAM cells may include 3D XPoint memory cells.

[0058] 5-7 show examples of systems 500, 600, and 700, respectively, and separate memories, each system including the memory chip 402 shown in FIG.

[0059] 5, bus 202 connects system 400 (including memory chip 402 and accelerator chip 404) to memory 204. Memory 204 (e.g., NVRAM) is a memory separate from the memory of first memory chip 402 of system 400. And, memory 204 can be a main memory.

[0060] In the system 500, the SoC 406 of the system 400 is connected to the memory 204 via the bus 202. And, the system 400 as a part of the system 500 includes a first memory chip 402, an accelerator chip 404, and the SoC 406. These parts of the system 400 are connected to the memory 204 via the bus 202. Also, as shown in FIG. 5, the memory controller 206 included in the SoC 406 controls data access of the memory 204 by the SoC 406 of the system 400. For example, the memory controller 206 controls data access of the memory 204 by the GPU 408 and / or the main processor 110. In some embodiments, the memory controller 206 can control data access of all memories in the system 500 (such as data access of the first memory chip 402 and the memory 204). And, the memory controller 206 can be communicatively coupled to the first memory chip 402 and / or the memory 204.

[0061] The memory 204 is separate from the memory provided by the first memory chip 402 of the system 400, and it can be used as memory by the GPU 408 and main processor 110 of the SoC 406 via the memory controller 206 and the bus 202. The memory 204 can also be used by the GPU 408 and the main processor 110 as memory for non-application specific or application specific tasks (such as non-AI or AI tasks) that are not executed by the accelerator chip 404. Data for such tasks can be accessed by and communicated to and from the memory 204 via the memory controller 206 and the bus 202.

[0062] In some embodiments, the memory 204 is a main memory of a device, such as the device that hosts the system 500. For example, in the system 500, the memory 204 can be the main memory 808 shown in FIG.

[0063] In FIG. 6, similar to FIG. 5, bus 202 connects system 400 (including memory chip 402 and accelerator chip 404) to memory 204. Unique to system 600 with respect to systems 500 and 700, first memory chip 402 includes a single set of pins 602 that directly connects first memory chip 402 to both accelerator chip 404 and SoC 406 via wires 614 and 616, respectively. Also as shown, in system 600, accelerator chip 404 includes a single set of pins 604 that directly connects accelerator chip 404 to first memory chip 402 via wires 614. Additionally, in system 600, the GPU of the SoC includes a set of pins 606 that directly connects SoC 406 to first memory chip 402 via wires 606.

[0064] In the system 600, the SoC 406 of the system 400 is connected to the memory 204 via the bus 202. And the system 400 as a part of the system 600 includes a first memory chip 402, an accelerator chip 404, and the SoC 406. These parts of the system 400 are connected to the memory 204 via the bus 202 (e.g., the accelerator chip 404 and the first memory chip 402 have an indirect connection to the memory 204 via the SoC 406 and the bus 202, and the SoC 406 has a direct connection to the memory 204 via the bus 202). Also, as shown in FIG. 6, the memory controller 206 included in the SoC 406 controls data access to the memory 204 by the SoC 406 of the system 400. For example, the memory controller 206 controls data access to the memory 204 by the GPU 408 and / or the main processor 110. In some embodiments, memory controller 206 can control data access of all memories in system 600 (e.g., data access of first memory chip 402 and memory 204), and memory controller 206 can be communicatively coupled to first memory chip 402 and / or memory 204.

[0065] The memory 204 is a separate memory (e.g., NVRAM) from the memory provided by the first memory chip 402 of the system 400, and it can be used as memory by the GPU 408 and main processor 110 of the SoC 406 via the memory controller 206 and the bus 202. The memory 204 can also be used by the GPU 408 and the main processor 110 as memory for non-application specific or application specific tasks (such as non-AI or AI tasks) that are not executed by the accelerator chip 404. Data for such tasks can be accessed by and communicated to and from the memory 204 via the memory controller 206 and the bus 202.

[0066] In some embodiments, memory 204 is a main memory of a device, such as the device that hosts system 600. For example, in system 600, memory 204 can be main memory 808 shown in FIG.

[0067] 7, bus 202 connects system 400 (including memory chip 402 and accelerator chip 404) to memory 204. Also, in system 700, bus 202 not only connects first memory chip 402 to SoC 406, but also connects first memory chip 402 to memory 204. As also shown, in system 700, bus 202 replaces second set of pins 416 of first memory chip 402, and wiring 426 and set of pins 417 of SoC 406 and GPU 408. First memory chip 402 in system 700 connects accelerator chip 404 and SoC 406 of system 400, similar to systems 500 and 600, except that this connection is via first set of pins 414 and bus 202.

[0068] Also, similar to systems 500 and 600, in system 700, memory 204 is a separate memory from the memory of the first memory chip 402 of system 400. In system 700, SoC 406 of system 400 is connected to memory 204 via bus 202. And in system 700, system 400 as a part of system 700 includes first memory chip 402, accelerator chip 404, and SoC 406. These parts of system 400 are connected to memory 204 via bus 202 in system 700. Also, similar to that shown in FIG. 7, memory controller 206 included in SoC 406 controls data access of memory 204 by SoC 406 of system 400. In some embodiments, memory controller 206 can control data access of all memories in system 700 (such as data access of first memory chip 402 and memory 204). And memory controller 206 can be communicatively coupled to first memory chip 402 and / or memory 204.

[0069] Also in system 700, memory 204 is a separate memory (e.g., NVRAM) from the memory provided by first memory chip 402 of system 400, which can be used as memory by GPU 408 and main processor 110 of SoC 406 via memory controller 206 and bus 202. Additionally, accelerator chip 404 can use memory 204 via first memory chip 402 and bus 202 in some embodiments and circumstances. In such examples, first memory chip 402 can include a cache for accelerator chip 404 and memory 204. Memory 204 can then be used by GPU 408 and main processor 110 as memory for non-application specific or application specific tasks (e.g., non-AI or AI tasks) not executed by accelerator chip 404. Data for such tasks can be accessed by and communicated to memory 204 via memory controller 206 and / or bus 202.

[0070] In some embodiments, the memory 204 is a main memory of a device, such as the device that hosts the system 700. For example, in the system 700, the memory 204 can be the main memory 808 shown in FIG.

[0071] Embodiments of accelerator chips disclosed herein (see, e.g., accelerator chip 102 and accelerator chip 404 shown in FIGS. 1-3 and 4-7, respectively) can be microprocessor chips or SoCs, etc. Accelerator chip embodiments can be designed for hardware acceleration of AI applications, including artificial neural networks, machine vision, and machine learning. In some embodiments, the accelerator chip (e.g., AI accelerator chip) can be configured to perform numerical calculations on vectors and matrices. In such embodiments, the accelerator chip can include a vector processor that performs numerical calculations on vectors and matrices (see, e.g., vector processors 112 and 412 shown in FIGS. 1-3 and 4-7, respectively, which can be configured to perform numerical calculations on vectors and matrices).

[0072] The accelerator chip embodiments disclosed herein may be or include an ASIC or FPGA. In ASIC embodiments of the accelerator chip, the accelerator chip is hardwired and dedicated to accelerating application-specific computations (such as AI computations). In some other embodiments, the accelerator chip may be a modified FPGA or GPU that has been modified for accelerating application-specific computations (such as AI computations) beyond an unmodified FPGA or GPU. In some other embodiments, the accelerator chip may be an unmodified FPGA or GPU.

[0073] An ASIC as described herein may include an IC that is customized for a particular use or application, such as accelerating application-specific computations (such as AI computations), as opposed to the general-purpose applications typically implemented by a CPU or another type of general-purpose processor, such as a GPU, which is typically for graphics processing.

[0074] The FPGAs described herein can be included in an IC that is designed and / or configured after the IC and FPGA are manufactured. Thus, the IC and FPGA are field programmable. The FPGA configuration can be specified using a Hardware Description Language (HDL). Similarly, the ASIC configuration can be specified using an HDL.

[0075] The GPUs described herein include ICs that can be configured to rapidly manipulate and modify memory to speed up the generation and updating of images in a frame buffer that is output to a display device. And, the systems described herein can include a display device connected to the GPU, and a frame buffer connected to the display device and the GPU. The GPUs described herein can be part of an embedded system, a mobile device, a personal computer, a workstation, or a game console, or any device that is connected to and uses a display device.

[0076] Each of the microprocessor chip embodiments described herein is one or more integrated circuits that incorporate at least the functionality of a central processing unit. Each microprocessor chip is multipurpose and may include at least a clock and registers that implement the chip by accepting binary data as input and using the registers and clock to process the data according to instructions stored in a memory connected to the microprocessor chip. Upon processing the data, the microprocessor chip may provide as output the results of the inputs and instructions. This output may then be provided to a memory connected to the microprocessor chip.

[0077] Each of the embodiments of the SoC described herein is one or more integrated circuits that integrate components of a computer or other electronic system. In some embodiments, the SoC is a single IC. In other embodiments, the SoC can include separate and connected integrated circuits. In some embodiments, the SoC can include its own CPU, memory, input / output ports, secondary storage, or any combination thereof. One or more such parts can be on a single substrate or microprocessor chip in the SoC described herein. In some embodiments, the SoC is smaller than a quarter, nickel, or dime. Some embodiments of the SoC can be part of a mobile device (such as a smartphone or tablet computer), an embedded system, or a device in the Internet of Things. In general, the SoC is different from systems that have a motherboard-based architecture that separates components based on functionality and connects these components through a central interface circuit board.

[0078] The embodiments of memory chips described herein (e.g., see first memory chip 104 shown in FIGS. 1-3 or first memory chip 402 shown in FIGS. 4-7) that are directly connected to an accelerator chip (e.g., an AI accelerator chip) are also referred to herein as application-specific memory chips for clarity when describing multiple memory chips across a system. The application-specific memory chips described herein are not necessarily hardwired for dedicated application-specific computation (e.g., AI computation). Each of the application-specific memory chips can be a DRAM chip or a NVRAM chip, or a memory device having similar functionality to either a DRAM chip or a NVRAM chip. Each of the application-specific memory chips can then be directly connected to an accelerator chip (e.g., an AI accelerator chip, see, e.g., accelerator chip 102 shown in Figures 1-3 and accelerator chip 404 shown in Figures 4-7) and can include memory units or cells dedicated to accelerating application-specific computations (such as AI computations) by the accelerator chip after the application-specific memory chip is configured by the accelerator chip or a separate SoC or processor (see, e.g., SoCs 106 and 406 shown in Figures 1-3 and 4-7, respectively).

[0079] The DRAM chips described herein may include random access memories that store each bit of data in a memory cell or unit having a capacitor and a transistor (such as a MOSFET). The DRAM chips described herein may take the form of an IC chip and may include billions of DRAM memory units or cells. In each unit or cell, the capacitor can either be charged or discharged. This can provide two states that are used to represent two values ​​in a bit. Because the charge on the capacitor can slowly leak out of the capacitor, an external memory refresh circuit is required to periodically rewrite the data in the capacitor to maintain the state of the capacitor and memory unit. DRAM is also a volatile memory in that it loses its data as soon as the power is removed, as opposed to a non-volatile memory such as flash memory or NVRAM. An advantage of DRAM chips is that they can be used in digital electronic devices that require large amounts of computer memory at low cost. DRAM also lends itself to being used as a main memory or memory dedicated to a GPU.

[0080] The NVRAM chips described herein may include non-volatile random access memory, which is a primary differentiating feature from DRAM. One example of an NVRAM unit or cell that may be used in the embodiments described herein may include a 3D XPoint unit or cell, in which bit storage is based on bulk resistance changes in combination with a stackable cross-lattice data access array.

[0081] The embodiments of the SoC described herein may include a main processor (e.g., a CPU or a main processor including a CPU). See, for example, the SoC 106 shown in FIGS. 1-3 and the SoC 406 shown in FIGS. 4-7, as well as the main processor 110 shown in FIGS. 1-7. In such embodiments, the GPU in the SoC (e.g., see the GPU 108 shown in FIGS. 1-3 and the GPU 408 shown in FIGS. 4-7) may execute instructions for application-specific tasks and calculations (e.g., AI tasks and calculations), and the main processor may execute instructions for non-application-specific tasks and calculations (e.g., non-AI tasks and calculations). And, in such embodiments, an accelerator chip connected to the SoC (e.g., see any one of the accelerator chips shown in FIGS. 1-7) may provide acceleration of application-specific tasks and calculations (e.g., AI tasks and calculations) dedicated to the GPU. Each of the embodiments of the SoC described herein may include its own bus for interconnecting the components of the SoC (e.g., connecting the main processor and the GPU). In addition, the bus of the SoC can be configured to connect the SoC to a bus external to the SoC so that components of the SoC can be coupled to chips and devices external to the SoC, such as another memory or memory chip (see, e.g., memory 204 shown in Figures 2-3 and Figures 5-7, and main memory 808 shown in Figures 8-9).

[0082] Non-application-specific computations and tasks of the GPU (e.g., non-AI computations and tasks), or application-specific computations and tasks that do not use the accelerator chip (e.g., AI computations and tasks), may not be traditional tasks performed by the main processor, but may use a separate memory (which may be application-specific memory), such as a separate memory chip, which may be implemented by DRAM, NVRAM, flash memory, or any combination thereof. See, for example, the memory 204 shown in FIGS. 2-3 and 5-7, as well as the main memory 808 shown in FIGS. 8-9. The separate memory or memory chip may be connected to the SoC and the main processor (e.g., CPU) via a bus external to the SoC (see, for example, the memory 204 shown in FIGS. 2-3 and 5-7, as well as the main memory 808 shown in FIGS. 8-9, and the bus 202 shown in FIGS. 2-3 and 5-7, as well as the bus 804 shown in FIGS. 8-9). In such embodiments, the separate memory or memory chip may have a memory unit dedicated to the main processor. Also, the separate memory or memory chips may be connected to the SoC and the GPU via a bus external to the SoC. In such embodiments, the separate memory or memory chips may include memory units or cells in the main processor or the GPU.

[0083] For purposes of this disclosure, it should be understood that each of the application specific memories or memory chips described herein (e.g., see first memory chip 104 shown in FIGS. 1-3 or first memory chip 402 shown in FIGS. 4-7) and the separate memories or memory chips described herein (e.g., see memory 204 shown in FIGS. 2-3 and 5-7, as well as main memory 808 shown in FIGS. 8-9) can be replaced with a group of memory chips, such as a memory chip string (e.g., see memory chip strings shown in FIGS. 10 and 11). For example, the separate memories or memory chips can be replaced with a memory chip string including at least an NVRAM chip and a flash memory chip downstream of the NVRAM chip. Also, the separate memory chips can be replaced with at least two memory chips, one of which is for the main processor (e.g., CPU) and another chip is for the GPU to use as memory for non-AI computations and / or tasks.

[0084] The embodiments of memory chips described herein can be part of a main memory and / or can be computer hardware that stores information for immediate use in a computer or for immediate use by any one of the processors described herein (e.g., any SoC or accelerator chip described herein). The memory chips described herein can operate faster than computer storage, which allows slower access to information but can also provide greater capacity and more reliable data. The memory chips described herein can include RAM, which is a type of memory that can have high operating speeds. The memory can be composed of addressable semiconductor memory units or cells, which can be implemented at least in part by MOSFETs.

[0085] Additionally, at least some embodiments disclosed herein relate to accelerator chips (e.g., AI accelerator chips) having vector processors (see, e.g., vector processors 112 and 412 shown in FIGS. 1-3 and 4-7, respectively), and at least some embodiments disclosed herein relate to forming memories using memory hierarchies and strings of memory chips (see, e.g., FIGS. 10 and 11).

[0086] Each embodiment of the vector processor described herein is an IC, and each IC can implement an instruction set that includes instructions that operate on one-dimensional arrays of data, called vectors, or multi-dimensional arrays of data, called matrices. Vector processors differ from scalar processors in that instructions operate on single data items. In some embodiments, vector processors can pipeline the data itself, rather than just pipeline the instructions. Pipelining can include the process in which the instructions, or in the case of vector processors, the data itself, are passed through multiple subunits in sequence. In some embodiments, vector processors are supplied with instructions that command arithmetic operations on vectors or matrices of numbers simultaneously. Instead of having to successively decode instructions and then fetch the data needed to complete those instructions, the vector processor reads a single instruction from memory, and the definition of the instruction itself simply implies that the instruction will operate again on another data item at an address one increment greater than the last. This can result in significant savings in decode time.

[0087] FIG. 8 illustrates an example of a parts arrangement of an exemplary computing device 800 according to some embodiments of the present disclosure. Examples of parts arrangements of the computing device 800 may include the system 100 shown in FIG. 1, the system 200 shown in FIG. 2, the system 400 shown in FIG. 4, the system 500 shown in FIG. 5, and the system 600 shown in FIG. 6. In the computing device 800, the application-specific components (see, for example, the application-specific components 807 in FIG. 8) may be AI components and may include the first memory chip 104 or 402 and the accelerator chip 102 or 404 shown arranged in FIG. 1, 2, 4, 5, and 6, respectively, as well as the SoC 106 or 406 shown arranged in FIG. 1, 2, 4, 5, and 6, respectively. In the computing device 800, wiring directly connects components of the application-specific components to each other (see, for example, the wiring 124 and 424 and the wiring 614 shown in FIG. 1-2 and FIG. 4-6, respectively). And, in computing device 800, wiring directly connects application-specific components to the SoC (see, e.g., wiring 817, which directly connects application-specific components to SoC 806). Wiring directly connecting application-specific components to the SoC may include wiring 126 as shown in Figures 1 and 2, or wiring 426 as shown in Figures 4 and 5. Wiring directly connecting application-specific components to the SoC may also include wiring 616 as shown in Figure 6.

[0088] The computing device 800 can be communicatively coupled to other computing devices via a computer network 802 as shown in Figure 8. The computing device 800 includes at least a bus 804 (which can be one or more buses, such as a combination of a memory bus and a peripheral bus), a SoC 806 (which can be or include a SoC 106 or 406), application specific components 807 (which can be an accelerator chip 102 and a first memory chip 104 or a first memory chip 402 and an accelerator chip 404), and a main memory 808 (which can be or include a memory 204), as well as a network interface 810 and a data storage system 812. The bus 804 communicatively couples the SoC 806, the main memory 808, the network interface 810, and the data storage system 812. The bus 804 can then include a point-to-point memory connection, such as the bus 202, and / or the wires 126, 426, or 616. Computing device 800 includes a computer system including at least one or more processors in a SoC 806, a main memory 808 (e.g., read-only memory (ROM), flash memory, DRAM such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), NVRAM, SRAM, etc.), and a data storage system 812, which communicate with each other via a bus 804 (which may include one or more buses and wiring).

[0089] The main memory 808 (which may be, include, or be included in the memory 204) may include the memory string 1000 shown in Figure 10. The main memory 808 may also include the memory string 1100 shown in Figure 11. In some embodiments, the data storage system 812 may include the memory string 1000 or the memory string 1100.

[0090] The SoC 806 may include one or more general-purpose processing devices, such as a microprocessor, a CPU, and the like. The SoC 806 may also include one or more special-purpose processing devices, such as a GPU, an ASIC, an FPGA, a digital signal processor (DSP), a network processor, a processor-in-memory (PIM), and the like. The SoC 806 may be one or more processors having a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The processors of the SoC 806 may be configured to execute instructions to perform the operations and steps discussed herein. The SoC 806 may further include a network interface device, such as a network interface 810, for communicating over one or more communication networks (e.g., the network 802).

[0091] The data storage system 812 may include a machine-readable storage medium (also known as a computer-readable medium) on which one or more sets of instructions or software embodying any one or more of the methods or functions described herein are stored. The instructions may reside, completely or at least partially, in the main memory 808 and / or in one or more of the processors of the SoC 806 during execution by the computer system. The instructions may reside, completely or at least partially, in the main memory 808 and / or in one or more of the processors of the SoC 806 during execution by the computer system. 506 constitutes a machine-readable storage medium.

[0092] Although the memory, processor, and data storage portions are each shown as single portions in the exemplary embodiment, each portion should be construed to include a single portion or multiple portions capable of storing instructions and performing their respective operations. Additionally, the term "machine-readable storage medium" should be construed to include any medium capable of storing or encoding a set of instructions for execution by a machine, causing the machine to perform any one or more of the methods of the present disclosure. Thus, the term "machine-readable storage medium" should be construed to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0093] FIG. 9 illustrates another example of a parts arrangement of an exemplary computing device 900 according to some embodiments of the present disclosure. Examples of parts arrangements of the computing device 900 may include the system 300 illustrated in FIG. 3 and the system 700 illustrated in FIG. 7. In the computing device 900, the application-specific components (see, for example, the application-specific components 807 in FIG. 9) may be AI components and may include the first memory chip 104 or 402 and the accelerator chip 102 or 404 as illustrated and arranged in FIG. 3 and 7, respectively, as well as the SoC 106 or 406 as illustrated and arranged in FIG. 3 and 7, respectively. In the computing device 900, wiring directly connects the components of the application-specific components to each other (see, for example, the wiring 124 and 424 illustrated in FIG. 3 and 7, respectively). However, in the computing device 900, wiring does not directly connect the application-specific components to the SoC. Alternatively, in computing device 900, one or more buses connect application-specific components to the SoC (see, for example, bus 804 shown configured in FIG. 9 and bus 202 shown configured in FIGS. 3 and 7).

[0094] As shown in Figures 8 and 9, devices 800 and 900 include many similar components. Computing device 900 can be communicatively coupled to other computing devices via a computer network 802 as shown in Figure 9. Similarly, as shown in Figure 9, computing device 900 includes at least a bus 804 (which can be one or more buses, such as a combination of a memory bus and a peripheral bus), a SoC 806 (which can be or include SoC 106 or 406), application specific components 807 (which can be accelerator chip 102 and first memory chip 104 or first memory chip 402 and accelerator chip 404), and a main memory 808 (which can be or include memory 204), as well as a network interface 810 and a data storage system 812. Similarly, bus 804 communicatively couples SoC 806, main memory 808, network interface 810, and data storage system 812. And, bus 804 may include bus 202 and / or point-to-point memory connections such as wires 126 , 426 , or 616 .

[0095] As mentioned above, at least some embodiments disclosed herein relate to forming memories using memory hierarchies and strings of memory chips.

[0096] 10 and 11 show examples of memory chip strings 1000 and 1100, respectively, that can be used in the separate memories (ie, memory 204) shown in FIGS. 2-3 and 5-7.

[0097] 10, memory chip string 1000 includes a first memory chip 1002 and a second memory chip 1004. First memory chip 1002 is directly wired to second memory chip 1004 (see, e.g., wire 1022) and configured to directly interact with the second memory chip. Each chip in memory chip string 1000 may include one or more sets of pins (see, e.g., sets of pins 1012 and 1014) for connecting to upstream and / or downstream chips in the string. In some embodiments, each chip in memory chip string 1000 may include a single IC enclosed in an IC package.

[0098] 10, a set of pins 1012 is part of a first memory chip 1002 and connects the first memory chip 1002 to a second memory chip 1004 via wires 1022 and a set of pins 1014 that is part of a second memory chip 1004. Wires 1022 connect the two sets of pins 1012 and 1014.

[0099] In some embodiments, the second memory chip 1004 can have the lowest memory bandwidth of the chips in the string 1000. In these and other embodiments, the first memory chip 1002 can have the highest memory bandwidth of the chips in the string 1000. In some embodiments, the first memory chip 1002 is or includes a DRAM chip. In some embodiments, the first memory chip 1002 is or includes a NVRAM chip. In some embodiments, the second memory chip 1004 is or includes a DRAM chip. In some embodiments, the second memory chip 1004 is or includes a NVRAM chip. And, in some embodiments, the second memory chip 1004 is or includes a flash memory chip.

[0100] 11, memory chip string 1100 includes a first memory chip 1102, a second memory chip 1104, and a third memory chip 1106. The first memory chip 1102 is directly wired to the second memory chip 1104 (see, e.g., wiring 1122) and configured to directly interact with the second memory chip. The second memory chip 1104 is directly wired to the third memory chip 1106 (see, e.g., wiring 1124) and configured to directly interact with the third memory chip. In such a manner, the first and third memory chips 1102 and 1106 interact with each other indirectly via the second memory chip 1104.

[0101] Each chip in memory chip string 1100 may include one or more sets of pins for connecting to upstream and / or downstream chips in the string (see, for example, sets of pins 1112, 1114, 1116, and 1118). In some embodiments, each chip in memory chip string 1100 may include a single IC enclosed in an IC package.

[0102] 11, a set of pins 1112 is part of the first memory chip 1102 and connects the first memory chip 1102 to the second memory chip 1104 via wires 1122 and a set of pins 1114 that is part of the second memory chip 1104. Wires 1122 connect the two sets of pins 1112 and 1114. Also, a set of pins 1116 is part of the second memory chip 1104 and connects the second memory chip 1104 to the third memory chip 1106 via wires 1124 and a set of pins 1118 that is part of the third memory chip 1106. Wires 1124 connects the two sets of pins 1116 and 1118.

[0103] In some embodiments, the third memory chip 1106 can have the lowest memory bandwidth of the chips in the string 1100. In these and other embodiments, the first memory chip 1102 can have the highest memory bandwidth of the chips in the string 1100. In these and other embodiments, the second memory chip 1104 can have the next highest memory bandwidth of the chips in the string 1100. In some embodiments, the first memory chip 1102 is or includes a DRAM chip. In some embodiments, the first memory chip 1102 is or includes a NVRAM chip. In some embodiments, the second memory chip 1104 is or includes a DRAM chip. In some embodiments, the second memory chip 1104 is or includes a NVRAM chip. In some embodiments, the second memory chip 1104 is or includes a flash memory chip. In some embodiments, the third memory chip 1106 is or includes a NVRAM chip. And, in some embodiments, the third memory chip 1106 is or includes a flash memory chip.

[0104] In embodiments having one or more DRAM chips, the DRAM chips may include logic circuits for command and address decoding, as well as a memory unit array of DRAM. The DRAM chips described herein may also include a cache or buffer memory for incoming and / or outgoing data. In some embodiments, the memory units implementing the cache or buffer memory may be different from the DRAM units on the chip that hosts the cache or buffer memory. For example, the memory units implementing the cache or buffer memory on a DRAM chip may be SRAM memory units.

[0105] In embodiments having one or more NVRAM chips, the NVRAM chips may include logic circuits for command and address decoding, as well as an array of NVRAM memory units, such as 3D XPoint memory units. The NVRAM chips described herein may also include cache or buffer memory for incoming and / or outgoing data. In some embodiments, the memory units implementing the cache or buffer memory may be different from the NVRAM units on the chip that hosts the cache or buffer memory. For example, the memory units implementing the cache or buffer memory on the NVRAM chips may be SRAM memory units.

[0106] In some embodiments, the NVRAM chip can include a cross-point array of non-volatile memory cells. The cross-point non-volatile memory array can perform bit storage based on bulk resistance changes in combination with a stackable cross-grid data access array. Furthermore, in contrast to many flash-based memories, the cross-point non-volatile memory can perform in-place write operations, where the non-volatile memory cells can be programmed without the non-volatile memory cells being erased beforehand.

[0107] As described herein above, the NVRAM chips can be or include cross-point storage and memory devices (e.g., 3D XPoint memory). Cross-point memory devices use transistorless memory elements, each of which includes memory cells and selectors stacked together as a column. The columns of memory elements are connected via two vertical layers of wires, one layer above the columns of memory elements and another layer below the columns of memory elements. Each memory element can be individually selected at the intersection of one wire on each of the two layers. Cross-point memory devices are fast, non-volatile, and can be used as a unified memory pool for processing and storage.

[0108] In embodiments having one or more flash memory chips, the flash memory chip may include logic circuitry for command and address decoding, as well as an array of flash memory memory units, such as NAND-type flash memory units. The flash memory chips described herein may also include a cache or buffer memory for incoming and / or outgoing data. In some embodiments, the memory units implementing the cache or buffer memory may be different from the flash memory units on the chip that hosts the cache or buffer memory. For example, the memory units implementing the cache or buffer memory on a flash memory chip may be SRAM memory units.

[0109] Also, for example, an embodiment of a memory chip string may include DRAM to DRAM to NVRAM, or DRAM to NVRAM to NVRAM, or DRAM to flash memory to flash memory, although DRAM to NVRAM to flash memory may provide a more effective solution for memory chip strings that are flexibly provisioned as multi-tier memory.

[0110] Also, for purposes of this disclosure, it should be understood that DRAM, NVRAM, 3D XPoint memory, and flash memory are technologies for individual memory units, and that a memory chip for any one of the memory chips described herein may include logic circuitry for command and address decoding, and a memory unit array of DRAM, NVRAM, 3D XPoint memory, or flash memory. For example, a DRAM chip described herein includes logic circuitry for command and address decoding, and a memory unit array of DRAM. For example, a NVRAM chip described herein includes logic circuitry for command and address decoding, and a memory unit array of NVRAM. For example, a flash memory chip described herein includes logic circuitry for command and address decoding, and a memory unit array of flash memory.

[0111] Also, the memory chip for any one of the memory chips described herein may include a cache or buffer memory for incoming and / or outgoing data. In some embodiments, the memory unit implementing the cache or buffer memory may be different from the unit on the chip that hosts the cache or buffer memory. For example, the memory unit implementing the cache or buffer memory may be an SRAM memory unit.

[0112] In the foregoing specification, the disclosed embodiments have been described with reference to certain exemplary embodiments thereof. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the disclosed embodiments as set forth in the following claims. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

1. An accelerator chip; A memory chip; A system comprising: The memory chip includes: a single set of pins configured to connect to the accelerator chip and a system on chip (SoC) via wiring, the accelerator chip and the memory chip being directly connected to each other via one set of pins of the accelerator chip and a portion of the single set of pins of the memory chip, and the SoC and the memory chip being directly connected to each other via one set of pins of the SoC and another portion of the single set of pins of the memory chip; a plurality of first memory cells configured to store and provide computational input data received from the SoC via the other portion of the single set of pins, the computational input data being used by the accelerator chip as computational input; and Including, The system, wherein the SoC communicates with the accelerator chip indirectly via the memory chip, and the SoC does not communicate directly with the accelerator chip via a bus or wiring.

2. 2. The system of claim 1, wherein the accelerator chip is an artificial intelligence (AI) accelerator chip, and the first plurality of memory cells are configured to store and provide AI calculation input data received from the SoC via the other portion of the single set of pins, the AI ​​calculation input data being used by the AI ​​accelerator chip as AI calculation input.

3. The memory chip includes: a plurality of second memory cells configured to store and provide first computation output data received from the accelerator chip via the portion of the single set of pins, the first computation output data being obtained by the SoC or reused by the accelerator chip as computation input; The system of claim 1 , comprising:

4. 4. The system of claim 3, wherein the memory chip includes a plurality of third memory cells configured to store second computation output data received from the SoC via the other portion of the single set of pins, the second computation output data being obtained by the SoC.

5. 5. The system of claim 4, wherein the first plurality of memory cells, the second plurality of memory cells, and the third plurality of memory cells comprise dynamic random access memory (DRAM) cells.

6. 5. The system of claim 4, wherein the first plurality of memory cells, the second plurality of memory cells, and the third plurality of memory cells comprise non-volatile random access memory (NVRAM) cells.

7. The system of claim 6 , wherein the NVRAM cells comprise 3D XPoint memory cells.

8. 4. The system of claim 3, wherein the accelerator chip includes a vector processor, the vector processor configured to perform numerical calculations on vectors and matrices for the SoC using the first plurality of memory cells and the second plurality of memory cells as memory.

9. 10. The system of claim 8, wherein the accelerator chip comprises an application specific integrated circuit (ASIC), the ASIC including the vector processor and being dedicated hardwired to accelerate AI calculations via the vector processor.

10. 10. The system of claim 8, wherein the accelerator chip includes a field programmable gate array (FPGA), the FPGA including the vector processor and being dedicated hardwired to accelerate AI calculations via the vector processor.

11. An accelerator chip; A system on chip (SoC); A memory chip; A system comprising: The memory chip includes: a set of pins configured to connect to the accelerator chip via wiring, the accelerator chip and the memory chip being directly connected to each other via the set of pins of the accelerator chip, the set of pins of the memory chip, and the wiring, and the SoC and the memory chip being communicatively connected to each other via a bus; a plurality of first memory cells configured to store and provide computational input data received from the SoC via the bus, the computational input data being used by the accelerator chip as computational input; Including, The system, wherein the SoC communicates with the accelerator chip indirectly via the memory chip, and the SoC does not communicate directly with the accelerator chip via a bus or wiring.

12. The memory chip includes: a plurality of second memory cells configured to store and provide first computation output data received from the accelerator chip via the set of pins of the memory chip, the first computation output data being obtained by the SoC or reused by the accelerator chip as a first computation input; The system of claim 11 , comprising:

13. 13. The system of claim 12, wherein the SoC includes a graphics processing unit (GPU), and the accelerator chip is configured to perform and accelerate calculations for the GPU using the first plurality of memory cells and the second plurality of memory cells as memory.

14. 14. The system of claim 13, wherein the accelerator chip includes a vector processor, the vector processor configured to perform numerical calculations on vectors and matrices for the GPU using the first plurality of memory cells and the second plurality of memory cells as memory.

Images