NVM-based high-capacity neural network inference engine

Abstract

A system, method, and computer program product for a neural network inference engine are disclosed. The inference engine system may include a first memory and a processor in communication with the first memory. The processor may be configured to perform operations. The operations the processor is configured to perform may include fetching a first task with the first memory and sending the first task to the processor for processing the first task. The operations may further include prefetching a second task with the first memory while the processor is processing the first task. The operations may further include the first memory sending the second task to the processor upon completion of processing the first task. The operations may further include the processor processing the second task.

Description

[Technical Field]

[0001] The present disclosure relates generally to the field of memory, and more particularly to retrieving data from memory. [Background technology]

[0002] Neural networks place increasingly large demands on memory subsystems. This is especially true for deep neural networks (DNNs) given growing model sizes and datasets. Off-chip memory accesses are often energy-intensive, time-consuming, and require large packaging form factors. Thermal challenges can also impose constraints on memory systems and the memory technologies used within them. Summary of the Invention

[0003] Embodiments of the present disclosure include systems, methods, and computer program products for a neural network inference engine. In some embodiments of the present disclosure, an inference engine system may include a first memory and a processor in communication with the first memory. The processor may be configured to perform operations. The operations configured to be performed by the processor may include fetching a first task using the first memory and sending the first task to the processor for processing the first task. The operations may further include prefetching a second task using the first memory while the processor is processing the first task. The operations may further include the first memory sending the second task to the processor upon finishing processing the first task. The operations may further include the processor processing the second task.

[0004] The above summary is not intended to describe each illustrated embodiment or every implementation of the present disclosure.

[0005] The drawings included in this disclosure are incorporated into and form a part of this specification. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. The drawings merely illustrate particular embodiments and do not limit the present disclosure. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates a memory stack according to the present disclosure. [Figure 2] FIG. 1 illustrates a memory stack with integrated artificial intelligence, according to an embodiment of the present disclosure. [Figure 3a] FIG. 1 illustrates a memory system according to an embodiment of the present disclosure. [Figure 3b] FIG. 1 illustrates a timeline of tasks completed by a memory system according to an embodiment of the present disclosure. [Figure 4a] FIG. 1 illustrates a memory system according to an embodiment of the present disclosure. [Figure 4b] FIG. 1 illustrates a timeline of tasks completed by a memory system according to an embodiment of the present disclosure. [Figure 5a] FIG. 1 illustrates a memory system according to an embodiment of the present disclosure. [Figure 5b] FIG. 1 illustrates a timeline of tasks completed by a memory system according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates a memory system according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates a memory system according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates a cloud computing environment according to an embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates an abstraction model layer according to an embodiment of the present disclosure. [Figure 10]FIG. 1 is a high-level block diagram of an example computer system that may be used when implementing one or more of the methods, tools, and modules described herein, and any associated functionality, in accordance with embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] While the embodiments described herein are susceptible to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that the particular embodiments described are not to be considered in a limiting sense. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0008] The present disclosure relates generally to the field of digital memory, and more particularly to retrieving data from memory. Further aspects of the present disclosure will be apparent to those skilled in the art. Some of these aspects are further described below.

[0009] Embodiments of the present disclosure include systems, methods, and computer program products for a non-volatile memory-based high-capacity neural network inference engine. Some embodiments may be particularly useful in deep neural network applications. In some embodiments of the present disclosure, an inference engine system may include a first memory and a processor in communication with the first memory. The processor may be configured to perform operations. The operations configured to be performed by the processor may include fetching a first task using the first memory and sending the first task to the processor for processing the first task. The operations may further include prefetching a second task using the first memory while the processor is processing the first task. The operations may further include the first memory sending the second task to the processor upon completion of processing the first task. The operations may further include the processor processing the second task.

[0010] To aid in understanding the present disclosure, FIG. 1 illustrates a memory stack 100 according to an embodiment of the present disclosure. The memory stack 100 includes memory dies 110a, 110b, 110c, 110d, 110e, 110f, and 110g stacked in a memory layer 110. Vertical interconnects 112 connect the memory dies 110a, 110b, 110c, 110d, 110e, 110f, and 110g to a buffer die 120. The vertical interconnects 112 may be, for example, microbumps, pillars, or direct pad-to-pad bonds. The buffer die 120 may have one or more buffer segments 120a, 120b, and 120c. The memory stack 100 may be connected to another system (e.g., a memory chip) by a controlled collapse chip connection (C4) 130.

[0011] In some embodiments, memory dies 110a, 110b, 110c, 110d, 110e, 110f, and 110g can include one or more types of non-volatile memory. Memory dies 110a, 110b, 110c, 110d, 110e, 110f, and 110g can include, for example, high-density memory. Memory dies 110a, 110b, 110c, 110d, 110e, 110f, and 110g can include, for example, phase-change memory and / or magnetoresistive random-access memory (MRAM).

[0012] In some embodiments, it may be preferable for memory dies 110a, 110b, 110c, 110d, 110e, 110f, and 110g to use a memory type that has high endurance, can maintain data at high temperatures, and has low latency. In some embodiments, memory tier 110 can contain read-only data, or memory tier 110 can contain data that is infrequently written or modified. For example, in some embodiments, a fully developed artificial intelligence (AI) model may be stored on memory tier 110; the AI ​​model may be fetched in whole or in part(s) from memory tier 110 by one or more buffer segments 120a, 120b, and 120c for use by the processor. Thus, a large AI model may be stored in memory stack 100 composed of slower, higher-density memory, and the latency of using the AI ​​model may be reduced by using faster, lower-density memory as buffer segments 120a, 120b, and 120c.

[0013] The buffer die 120 may include one or more buffers. In some embodiments, one buffer may be partitioned into multiple buffer segments 120a, 120b, and 120c. In some embodiments, multiple buffers may be used, and each buffer may be unpartitioned or may be partitioned into multiple buffer segments 120a, 120b, and 120c. The multiple buffer segments 120a, 120b, and 120c may perform multiple fetches simultaneously, serially, or some combination thereof.

[0014] Buffer segments 120a, 120b, and 120c may be memory. In some embodiments, buffer segments 120a, 120b, and 120c may be high-speed, low-density memory, such as static random-access memory (SRAM) or dynamic random-access memory (DRAM). In some embodiments, buffer segments 120a, 120b, and 120c may hold data while the processor processes other data, may hold data that the processor is processing, and / or may hold data while the processor processes data and rewrites the data (e.g., overwriting retrieved data with computations). In some embodiments, buffer segments 120a, 120b, and 120c can send processed data (e.g., task computation results) to memory layer 110 for storage; in some embodiments, buffer segments 120a, 120b, and 120c can send processed data to memory layer 110 for storage, and then fetch other data and send it to the processor for processing and / or computation.

[0015] In some embodiments of the present disclosure, the memory stack 100 system can generate a first task computation result in response to a processor processing a first task. The processor can transmit the first task computation result to a first memory (e.g., buffer or buffer segment 120a, 120b, and 120c); the first memory can accept the first task computation result. The first memory can send the first task computation result to a second memory (e.g., memory stack 100 or buffer or memory die 110a, 110b, 110c, 110d, 110e, 110f, or 110g). In some embodiments, the first memory can be a low-density memory and the second memory can be a high-density memory. In some embodiments, the second memory may be integrated into a three-dimensional stack of memory (e.g., a memory stack), the three-dimensional stack of memory comprising multiple memory layers (e.g., memory dies 110a, 110b, 110c, 110d, 110e, 110f, and 110g).

[0016] In some embodiments of the present disclosure, the first memory is a buffer, the buffer is in a buffer die, and the buffer die has an artificial intelligence core. Figure 2 illustrates an AI integrated memory stack 204 and its components, according to an embodiment of the present disclosure.

[0017] The memory stack 200 may include a memory die layer 210 including one or more memory dies 210a, 210b, and 210c. The memory stack 200 may also include a buffer die 220 including one or more buffer segments 220a, 220b, and 220c. The memory stack 200 may be combined with an AI unit 202. The AI ​​unit 202 may include an AI accelerator 232 having an AI core set 234. The AI ​​core set may include multiple AI cores 234a, 234b, 234c, and 234d. One or more of the AI ​​cores may include, for example, a scratchpad (e.g., a digital scratchpad and / or scratchpad memory); in some embodiments, the scratchpad may include multiple buffering (e.g., double buffering) to reduce or hide latency. AI accelerator 232 may be integrated within buffer die 220 to form AI-enabled buffer die 250. In some embodiments, AI-enabled buffer die 250 may include AI acceleration as well as compute capabilities (e.g., it may include a processor).

[0018] In some embodiments, the buffer die and the compute die may be combined. Figure 3a illustrates a memory system 300 according to such an embodiment of the present disclosure. A memory 310 is shown in communication with a compute die 320. A buffer 326 within the compute die 320 has fetched task A 324a and sends it to a compute core 328. Meanwhile, a prefetch controller 322 directs the prefetching of task B 324b from memory 310.

[0019] 3b illustrates a graphical timeline 340 of tasks completed by memory system 300, according to an embodiment of the present disclosure. Graphical timeline 340 illustrates tasks completed over time 342 (x-axis). The work of each component of memory system 300 is identified by the component's name (listed on the y-axis), and the tasks the component performs are shown on the graph.

[0020] The buffer 326 performs buffer operations 352, which may also be referred to as work completed by the buffer. The buffer operations 352 may include fetching (or prefetching) and containing data 352a for task A. The buffer operations 352 may also include prefetching (or fetching) and containing data 352b for task B. The compute core 328 performs core operations 356, which may also be referred to as work completed by the compute core. The core operations 356 include performing task A 356a and performing task B 356b. The prefetch controller 322 performs prefetch controller operations 358, which may also be referred to as work performed by the prefetch controller. The prefetch controller operations 358 may include triggering prefetching 358a of data for task B and triggering prefetching 358b of data for task C.

[0021] Data 352a for task A may be held in buffer 326 before, during, or after compute core 328 performs task A 356a, or a combination thereof. In some embodiments, compute core 328 may receive data 356a for task A from buffer 326 for computation, thereby making buffer 326 available for performing other tasks. In such embodiments, compute core 328 may send the results of completed task A 356a (e.g., processed data) to the same buffer 326 (or buffer segment) or a different buffer (or buffer segment) for transfer to storage memory (e.g., memory 310 or a different memory where the processed data can be used for another task).

[0022] In some embodiments, both the buffer 326 and the AI ​​accelerator may be integrated within the compute die 320. The AI ​​accelerator may have its own subunit within the compute die 320 or may be integrated within another component (e.g., the AI ​​accelerator may be integrated within the buffer 326).

[0023] In some embodiments, alternating operation (e.g., fetching data) between a first buffer (or segment) and a second buffer (or segment) may be used. Buffer unit alternation between buffers (or buffer segments) may be referred to as a ping-pong buffer. FIG. 4a illustrates a memory system 400 according to such an embodiment of the present disclosure.

[0024] The memory 410 is shown in communication with a compute die 420. A first buffer 426a (or buffer segment) in the compute die 420 has fetched task A 424a and sends it to a compute core 428, and a second buffer 426b in the compute die 420 has prefetched task B 424b and sends it to a compute core 428. A multiplexer (MUX) 430 can direct data traffic, and a prefetch controller 422 can direct the prefetching of task C 424c from the memory 410.

[0025] The MUX 430 can order and direct data from the first buffer 426a and the second buffer 426b. For example, the MUX 430 can first send task A 424a from the first buffer 426a to the compute core 428 and then send task B 424b from the second buffer 426b to the compute core 428. The MUX 430 can wait for one task to complete before sending another task. The MUX 430 can also direct data resulting from a task (e.g., processed data, such as a computation result, resulting from a task) to a buffer (e.g., the first buffer 426a, the second buffer 426b, or both), which can then send the data to a storage memory (e.g., the memory 410 or an external memory). Alternatively or additionally, MUX 430 can direct the buffer to send the data resulting from the completed task to a different processor (e.g., a compute core in a connected or separate system) that can use the data resulting from the task to compute other data; for example, the processed data can be forwarded to another system as input data for another computation.

[0026] A ping-pong buffer may contain one buffer (or buffer segment) from which a task is fetched while the processor is working on another task and therefore not yet ready to accept a new task. Fetching a task before the processor is ready to compute the task may be called prefetching.

[0027] 4b illustrates a graphical timeline 440 of tasks completed by memory system 400, according to an embodiment of the present disclosure. Graphical timeline 440 illustrates tasks completed over time 442 (x-axis). The work of each component of memory system 400 is identified by the component's name (listed on the y-axis), and the tasks the component performs are shown on the graph.

[0028] The first buffer 426a performs first buffer work 452, which may also be referred to as work completed by the first buffer. The first buffer work 452 may include fetching (or prefetching) and containing data 452a for task A. The first buffer work 452 may also include prefetching (or fetching) and containing data 452c for task C. The second buffer 426b performs second buffer work 454, which may also be referred to as work completed by the second buffer. The second buffer work 454 may include fetching (or prefetching) and containing data 454a for task B.

[0029] Compute core 428 performs core work 456, which may also be referred to as work completed by a compute core. Core work 456 includes performing task A 456a, performing task B 456b, and performing task C 456c. Prefetch controller 422 performs prefetch controller work 458, which may also be referred to as work performed by the prefetch controller. Prefetch controller work 458 may include triggering prefetching of data for task B 458a and triggering prefetching of data for task C 458b.

[0030] Data 452a for task A may be held in first buffer 426a before, during, or after compute core 428 performs task A 456a, or a combination thereof. In some embodiments, compute core 428 may receive data 456a for task A from first buffer 426 for computation, thereby making buffer 426 available for performing other tasks. In such embodiments, compute core 428 may send the results of completed task A 456a (e.g., processed data) to the same first buffer 426a (or buffer segment) or a different buffer (or buffer segment) for transfer to storage memory (e.g., memory 410 or a different memory where the processed data can be used for another task).

[0031] In some embodiments, ping-pong buffers may be preferable for lower latency, as realized by the end user. For example, memory 410 may be a slow, high-density memory that may require several seconds to retrieve data; interleaving allows compute core 428 to perform work on task A 452a in first buffer 426a while second buffer 426b prefetches task B 452b, such that task B 452b is retrieved and ready for compute core 428 to work on task B 452b immediately after completing task A 452a. In such an embodiment, first buffer 426a can send computation results to its destination and prefetch task C 424c.

[0032] In some embodiments of the present disclosure, the buffer may be a first memory having a first buffer segment and a second buffer segment, where the first buffer segment can fetch a first task and the second buffer segment can prefetch a second task, where the second buffer segment can fetch the second task while the processor is processing the first task.

[0033] In some embodiments, the buffer and compute cores may be on separate dies. Figure 5a illustrates a memory system 500 according to such an embodiment of the disclosure. Memory 510 is shown in communication with a buffer die 526 and a compute die 520. Buffer die 526 is in communication with compute die 520.

[0034] The buffer die 526 has a first buffer segment 526a and a second buffer segment 526b. Each buffer segment has fetched tasks that can be sent to a processor or compute core 528 for processing and / or computation. The first buffer segment 526a fetches task A 524a and sends it to the compute core 528, and the second buffer segment 526b fetches task B 524b and sends it to the compute core 528. Meanwhile, the prefetch controller 522 directs the prefetching of task C 524c from the memory 510.

[0035] A MUX (not shown) may be used to order data traffic and direct data traffic to and from buffer segments to compute cores 528. In some embodiments, the MUX may direct the buffer to return processed data (e.g., computation results) to the same memory 510; in some embodiments, the MUX may direct processed data to a different memory, another compute core, another processor, or other location or combinations thereof for storage and / or use.

[0036] 5b illustrates a timeline of tasks completed by a memory system according to such an embodiment of the present disclosure. Graphical timeline 540 illustrates tasks completed over time 542 (x-axis). The work of each component of memory system 500 is identified by the component's name (listed on the y-axis), and the tasks the component performs are shown on the graph.

[0037] Buffer 526 performs buffer operations 552, which may also be referred to as work completed by a buffer. Buffer operations 552 may include fetching (or prefetching) and containing data 552a for task A. Buffer operations 552 may also include prefetching (or fetching) and containing data 552b for task B. Compute core 528 performs core operations 556, which may also be referred to as work completed by a compute core. Core operations 556 include performing task A 556a and performing task B 556b. Prefetch controller 522 performs prefetch controller operations 558, which may also be referred to as work performed by the prefetch controller. Prefetch controller operations 558 may include triggering prefetching 558a of data for task B and triggering prefetching 558b of data for task C.

[0038] Data 552a for task A may be held in buffer 526 before, during, or after compute core 528 performs task A 556a, or a combination thereof. In some embodiments, compute core 528 may receive data 556a for task A from buffer 526 for computation, thereby making buffer 526 available for performing other tasks. In such embodiments, compute core 528 may send the results of completed task A 556a (e.g., processed data) to the same buffer 526 (or buffer segment) or a different buffer (or buffer segment) for transfer to storage memory (e.g., memory 510 or a different memory where the processed data can be used for another task).

[0039] In some embodiments, both the buffer 526 and the AI ​​accelerator may be integrated within the compute die 520. The AI ​​accelerator may have its own subunit within the compute die 520 or may be integrated within another component (e.g., the AI ​​accelerator may be integrated within the buffer 526).

[0040] In some embodiments of the present disclosure, the system may further include an error correction engine in communication with the memory and the processor. Figure 6 illustrates a memory system 600 according to such an embodiment of the present disclosure.

[0041] The memory 610 is shown in communication with a buffer 626 and a compute die 620. The buffer 626 is in communication with the compute die 620. The memory 610 can communicate data bits 622a to the compute die 620. This communication can occur directly (as shown) or through a buffer (not shown). The memory 610 can also communicate check bits 622b to the compute die 620. This communication can occur directly (as shown) or through a buffer (as shown). Figure 6 shows the check bits 622b traveling to the compute die 620 via the buffer 626 and the data bits 622a being sent directly from the memory 610 to the compute die 620.

[0042] The data bits 622 a and check bits 622 b are fed to an error correction engine 624. The data may be verified and, if desired, corrected before being sent to the compute cores 628. The error correction engine 624 may be located on the memory die, buffer die, compute die 620, or a different die (e.g., a different memory die or a dedicated error correction die). In some embodiments, the data bits 622 a, check bits 622 b, and error correction engine 624 may be co-located (e.g., on the same memory die, the same buffer die, the same compute die, or a separate error correction die).

[0043] In some embodiments of the present disclosure, sensors may be implemented to protect the system, the data the system contains, or one or more associated systems (e.g., data collection systems), or a combination thereof. For example, in some embodiments, the system may include a first temperature sensor for detecting a first temperature; the first temperature sensor may be in communication with a power gate, which throttles power if the first temperature threshold is reached. In some embodiments, the system may further include a second temperature sensor in communication with the power gate for detecting a second temperature. The power gate may throttle power if the second temperature threshold is reached. The power gate may be associated with any number of components and may throttle power to one or more of the components (e.g., just the buffer or the entire system) based on a temperature threshold. FIG. 7 illustrates a memory system 700 according to such an embodiment of the present disclosure.

[0044] Memory 710 is shown in communication with a prefetch controller 722, a buffer 726, and a power gate 730 on a buffer die 720. The buffer 726 is in communication with a compute core 728. The memory 710 can send programs to the compute core 728 via the buffer 726. Figure 7 shows a compute core 728 working on program A 724a and a buffer 726 containing program B 724b. The prefetch controller 722 is sending a prefetch request to the memory for program C 724c.

[0045] Memory 710 includes temperature sensors 732a, 732b, 732c, 732d, and 732e in communication with power gate 730. Similarly, buffer die 720 includes temperature sensors 734a and 734b in communication with power gate 730. Temperature sensors 732a, 732b, 732c, 732d, and 732e in memory 710 may be evenly distributed throughout memory 710 or may be concentrated in one or more leading regions (e.g., regions expected to reach a particular temperature or threshold most quickly). Similarly, temperature sensors 734a and 734b in buffer die 720 may be evenly distributed throughout buffer die 720 or may be concentrated in one or more leading regions (e.g., regions expected to reach a particular temperature or threshold most quickly). The number and placement of temperature sensors 732a, 732b, 732c, 732d, 732e, 734a, and 734b in system 700 may vary depending on the system, its components, risk tolerance, and user preferences.

[0046] Power gate 730 can throttle (e.g., limit, increase, or eliminate) the power supply to any, some, or all of the components in system 700. Power throttling can be the result of a manual command (e.g., a user command) or an automated command (e.g., reaching a threshold). An example of a manual command is when a user identifies a reason to turn off system 700 and manually instructs power gate 730 to interrupt power to system 700. An example of an automated command is when temperature sensor 734b indicates to power gate 730 that the temperature adjacent buffer 726 exceeds a safe temperature threshold, causing power gate 730 to reduce the power supplied to buffer 726.

[0047] In some embodiments, the threshold temperature may be consistent throughout the system. For example, power gate 730 may reduce power to the system or any component thereof if any of temperature sensors 732a, 732b, 732c, 732d, 732e, 734a, and 734b exceeds 90 degrees Celsius. In some embodiments, the threshold temperature may be set specifically for the associated component. For example, memory 710 may be able to tolerate (e.g., operate at) a higher temperature than buffer 726; in such a case, temperature sensors 732a, 732b, 732c, 732d, and 732e in memory 710 may have a threshold temperature of 120 degrees Celsius, while temperature sensors 734a and 734b in buffer die 720 may have a threshold temperature of 80 degrees Celsius. If different components are held within different areas of buffer die 720 and each component has a different temperature tolerance, the temperature thresholds of temperature sensors 734a and 734b may be different from one another. Similarly, if different components are held within different areas of memory die 710 and each component has a different temperature tolerance, the temperature thresholds of temperature sensors 732a, 732b, 732c, 732d, 732e, 734a, and 734b may be different from one sensor to another.

[0048] The thresholds may be set based on the materials that system 700 includes and how system 700 is constructed. For example, if the memory cells used in memory 710 can safely operate from 5° Celsius to 75° Celsius, power gate 730 may trigger power throttling if one or more of temperature sensors 732a, 732b, 732c, 732d, and 732e in memory 710 reaches 76° Celsius or drops below 5° Celsius. In some embodiments, the memory cells used in memory 710 may be selected for their enhanced thermal tolerance (e.g., the ability to operate safely at increased temperatures); if the memory cells used in memory 710 can safely operate from 5° Celsius to 125° Celsius, power gate 730 may trigger power throttling if one or more of temperature sensors 732a, 732b, 732c, 732d, and 732e in memory 710 reaches 126° Celsius or drops below 5° Celsius.

[0049] Similarly, temperature thresholds for buffer die 720 and other components may be set similarly. For example, a buffer 726 that safely operates from 0° Celsius to 50° Celsius may have throttling thresholds set to stop the delivery of power if either temperature sensor 734a or 734b receives a temperature reading above 50° Celsius or below 0° Celsius. Similarly, temperatures may also be sensed on or near compute cores 728, prefetch controller 722, power gates 730, and other components of system 700, and throttling thresholds may reflect the particular operating tolerances of the various components.

[0050] In some embodiments of the present disclosure, temperatures may be sensed and / or tracked in or near various components of system 700, and the throttling threshold temperature may be the same for all of the components. For example, a user may set a uniform throttling threshold temperature of 45° Celsius for any sensor in the system. The temperature threshold may be set automatically (e.g., the threshold is preset to be the same for every system), semi-automatically (e.g., system specifications are used to automatically load temperature thresholds for a particular system and / or component), manually (e.g., a user may input one or more temperature thresholds), or some combination thereof.

[0051] In some embodiments, the thresholds may be set based on the materials included in system 700, how system 700 is constructed, and the system's geometry. For example, a particular memory cell may be allowed to operate at a temperature above the standard temperature for up to three seconds, such that the integrity of the memory cell is maintained if the temperature falls within the standard temperature within three seconds. In such a case, memory cells located in a well-ventilated planar memory geometry may cool more quickly than memory cells in a memory stack (see FIG. 1); therefore, the temperature thresholds associated with temperature sensors 732a, 732b, 732c, 732d, and 732e in the planar memory geometry may trigger only after reaching an excessive temperature for more than three seconds, while temperature sensors 732a, 732b, 732c, 732d, and 732e in the memory stack may trigger immediately.

[0052] The present disclosure can be implemented in a variety of systems, including, but not limited to, on-site hard-wired memory storage, memory storage accessed via the cloud, analog storage, and digital storage. The present disclosure can enable faster memory access for any system that can use memory and compute; the memory and compute can communicate together and directly (e.g., part of one physical computer system), via a local connection (e.g., a local area network), via a private connection (e.g., a virtual private network), or via some other connection (e.g., a wide area network or the Internet).

[0053] Although this disclosure includes detailed descriptions of cloud computing, it is understood that implementation aspects of the teachings recited herein are not limited to cloud computing environments. Rather, embodiments of the present disclosure can be implemented in conjunction with any other type of computing environment now known or later developed.

[0054] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0055] Its features are as follows:

[0056] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server uptime and network storage, automatically as needed, without the need for human interaction with the provider of the service.

[0057] Broad Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0058] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge over the exact portion of the resources provided, although there is a sense of portion independence in that they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0059] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, sometimes automatically, to instantly scale out and rapidly released to instantly scale in. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0060] Services are measurable: Cloud systems automatically control and optimize resource usage by utilizing metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, available user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services used.

[0061] The service model is as follows:

[0062] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0063] Platform as a Service (PaaS) Service): The capability offered to a consumer is the deployment of applications that the consumer has created or acquired on a cloud infrastructure, where the applications are written using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but the consumer does have control over the deployed applications and possibly the application hosting environment configuration.

[0064] Infrastructure as a Service (IaaS): The capability provided to a consumer is the supply of processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, and deployed applications, and the consumer perhaps has limited control over certain networking components (e.g., host firewalls).

[0065] The deployment model is as follows:

[0066] Private Cloud: The cloud infrastructure operates solely for the organization. It can be managed by the organization or a third party and can exist on-premise or off-premise.

[0067] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policies, and / or regulatory compliance considerations). Cloud infrastructure can be managed by the organization or a third party and can exist on-premises or off-premises.

[0068] Public Cloud: Cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0069] Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public), each of which remains a unique entity but is bound together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0070] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0071] FIG. 8 illustrates a cloud computing environment 810 according to an embodiment of the present disclosure. As shown, the cloud computing environment 810 includes one or more cloud computing nodes 800 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cellular phone 800A, a desktop computer 800B, a laptop computer 800C, or an automobile computer system 800N, or combinations thereof, can communicate. The nodes 800 can communicate with each other. The nodes 800 may be physically or virtually grouped in one or more networks (not shown), such as private, community, public, or hybrid clouds, or combinations thereof, as described above.

[0072] This allows the cloud computing environment 810 to provide infrastructure-as-a-service, platform-as-a-service, and / or software-as-a-service services without the need for cloud consumers to maintain resources on local computing devices. It is understood that the types of computing devices 800A-N shown in Figure 8 are intended to be merely illustrative, and that computing nodes 800 and cloud computing environment 810 can communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0073] 9 illustrates abstraction model layers 900 provided by cloud computing environment 810 (of FIG. 8) according to an embodiment of the present disclosure. It should be understood in advance that the components, layers, and functions illustrated in FIG. 9 are intended to be merely illustrative, and that embodiments of the disclosure are not limited thereto. As shown below, the following layers and corresponding functions are provided:

[0074] The hardware and software layer 915 includes hardware and software components. Examples of hardware components include a mainframe 902, a RISC (Reduced Instruction Set Computer) architecture-based server 904, a server 906, a blade server 908, storage devices 911, and network and networking components 912. In some embodiments, the software components include network application server software 914 and database software 916.

[0075] The virtualization layer 920 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 922; virtual storage 924; virtual networks 926, including virtual private networks; virtual applications and operating systems 928; and virtual clients 930.

[0076] In one example, the management layer 940 can provide the functions described below. Resource provisioning 942 enables dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 944 provides price tracking as resources are utilized within the cloud computing environment and billing or invoicing is utilized for the consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and protection for tasks and data and other resources. User portal 946 provides access to the cloud computing environment for consumers and system administrators. Service level management 948 provides allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 950 enables pre-allocation and procurement of cloud computing resources for which future requirements are forecasted according to SLAs.

[0077] The workload layer 960 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 962, software development and lifecycle management 974, virtual classroom instructional delivery 966, data analytics processing 968, transaction processing 970, and one or more NVM-based high-capacity neural network inference engines 972.

[0078] Although this disclosure includes detailed descriptions of cloud computing, it is understood that implementations of the teachings recited herein are not limited to cloud computing environments. Rather, embodiments of the present disclosure can be implemented in conjunction with any other type of computing environment now known or that may later be developed.

[0079] 10 illustrates a high-level block diagram of an example computer system 1001 that may be used to implement (e.g., using one or more processor circuits or computer processors of a computer) one or more of the methods, tools, and modules, and any associated functionality, described herein, in accordance with embodiments of the present disclosure. In some embodiments, the major components of computer system 1001 may include a processor 1002 having one or more central processing units (CPUs) 1002A, 1002B, 1002C, and 1002D, a memory subsystem 1004, a terminal interface 1012, a storage interface 1017, an I / O (Input / Output) device interface 1014, and a network interface 1018, all of which may be communicatively coupled, directly or indirectly, for inter-component communication via a memory bus 1003, an I / O bus 1008, and an I / O bus interface unit 1010.

[0080] The computer system 1001 may include one or more general-purpose programmable CPUs 1002A, 1002B, 1002C, and 1002D, collectively referred to herein as CPUs 1002. In some embodiments, the computer system 1001 may include multiple processors, which is typical of larger systems; however, in other embodiments, the computer system 1001 may alternatively be a single-CPU system. Each CPU 1002 may execute instructions stored in a memory subsystem 1004 and may also include one or more levels of on-board cache.

[0081] System memory 1004 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 1022 or cache memory 1024. Computer system 1001 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1027 may be provided for reading from and writing to non-removable, non-volatile magnetic media, such as a "hard drive." Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), or an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. Additionally, memory 1004 may include flash memory, such as a flash memory stick drive or flash drive. Memory devices may be connected to memory bus 1003 by one or more data media interfaces. The memory 1004 may include at least one program product having a set of program modules (eg, at least one program module) configured to perform the functions of various embodiments.

[0082] One or more programs / utilities 1028, each having at least one set of program modules 1030, may be stored in memory 1004. The programs / utilities 1028 may include a hypervisor (also called a virtual machine monitor), one or more operating systems, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The programs 1028 and / or program modules 1030 generally implement the functions or methods of various embodiments.

[0083] 10 as a single bus structure providing a direct communication path between CPU 1002, memory subsystem 1004, and I / O bus interface 1010, memory bus 1003, in some embodiments, may include multiple different buses or communication paths, which may be arranged in any of a variety of configurations, such as point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. Additionally, I / O bus interface 1010 and I / O bus 1008 are shown as single respective units, and computer system 1001, in some embodiments, may include multiple I / O bus interfaces 1010, multiple I / O buses 1008, or both. Additionally, although multiple I / O bus interfaces 1010 are shown isolating the I / O bus 1008 from the various communication paths extending to the various I / O devices, in other embodiments, some or all of the I / O devices may be directly connected to one or more system I / O buses 1008.

[0084] In some embodiments, computer system 1001 may be a multi-user mainframe computer system, a single-user system, a server computer, or a similar device that has little or no direct user interface but receives requests from other computer systems (clients). Further, in some embodiments, computer system 1001 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, a network switch or router, or any other suitable type of electronic device.

[0085] It is noted that Figure 10 is intended to illustrate representative major components of example computer system 1001. In some embodiments, however, individual components may have greater or less complexity than those shown in Figure 10, components other than or in addition to those shown in Figure 10 may be present, and the number, type, and configuration of such components may vary.

[0086] The present disclosure may be a system, method, or computer program product, or a combination thereof, at any conceivable level of technical detail. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to execute aspects of the present disclosure.

[0087] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, or semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridge structures with instructions recorded on them, and any suitable combination of the above. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, or electrical signals transmitted through other transmission media (e.g., light pulses passing through fiber optic cable) or wires.

[0088] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, fiber optic transmission cables, wireless transmission cables, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0089] Computer-readable program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, or the like, and procedural programming languages ​​such as the “C” programming language or similar. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to implement aspects of the present disclosure.

[0090] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0091] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, the instructions of which, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium, and the computer-readable program instructions may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium in which the instructions are stored constitutes an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0092] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps performed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, such that the instructions, executed on the computer, other programmable data processing apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0093] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions described in the blocks may occur out of the order described in the figures. For example, two blocks shown in succession may in fact be performed as a single step, or may be performed simultaneously, substantially simultaneously, partially, or wholly in a time-overlapping manner, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that executes a combination of special-purpose hardware and computer instructions to perform the specified functions or operations or to execute a combination of special-purpose hardware instructions and computer instructions.

[0094] While the present disclosure has been described with reference to specific embodiments, it is expected that alternatives and modifications thereof will become apparent to those skilled in the art. The descriptions of various embodiments of the present disclosure have been presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will become apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein have been selected to best explain the principles of the invention, practical applications, or technical improvements over techniques found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein. It is therefore intended that the appended claims be construed to cover all such changes and modifications that fall within the scope of the present disclosure.

Claims

1. 1. An inference engine system, comprising: a first memory having a first buffer segment and a second buffer segment; a processor in communication with the first memory, the processor comprising: fetching a first task from a memory die of a non-volatile memory via a vertical interconnect using the first buffer segment; submitting the first task to the processor via the first buffer segment; prefetching a second task from the memory die via the vertical interconnect using the second buffer segment while the processor is processing the first task; Upon completion of processing the first task, submitting the second task to the processor via the second buffer segment; and processing the second task by the processor; 1. An inference engine system configured to perform operations including:

2. The operation is generating a first task computation result in response to the processor processing the first task; transmitting, by the processor, the first task calculation result to the first memory; receiving the first task calculation result by the first memory; and sending the first task calculation result to the memory die; 2. The system of claim 1, wherein the first memory is a lower density memory than the memory die.

3. 3. The system of claim 2, wherein the memory dies are integrated into a three-dimensional stack of memory, the three-dimensional stack of memory comprising multiple memory layers.

4. 4. The system of claim 1, further comprising a first temperature sensor in communication with a power gate for sensing a first temperature, the power gate throttling power to the first memory if a first temperature threshold is reached.

5. 5. The system of claim 4, further comprising a second temperature sensor in communication with the power gate for sensing a second temperature, the power gate throttling power to the first memory if a second temperature threshold is reached.

6. 6. The system of claim 1, further comprising an error correction engine in communication with the first memory and the processor, wherein data bits and check bits for the error correction engine are co-located.

7. the first memory is in a buffer die; The system of any one of claims 1 to 6, wherein the buffer die comprises an artificial intelligence core.

8. 1. A method for memory storage and retrieval, comprising: fetching a first task from a memory die of a non-volatile memory via a vertical interconnect using a first buffer segment of a first memory, the first memory having the first buffer segment and a second buffer segment; submitting the first task to a processor via the first buffer segment; prefetching a second task from the memory die via the vertical interconnect using the second buffer segment while the processor is processing the first task; Upon completion of processing the first task, submitting the second task to the processor via the second buffer segment; and processing the second task by the processor; A method comprising:

9. generating a first task computation result in response to the processor processing the first task; transmitting, by the processor, the first task calculation result to the first memory; receiving the first task calculation result by the first memory; and sending the first task calculation result to the memory die; 9. The method of claim 8, wherein the first memory is a lower density memory than the memory die.

10. 10. The method of claim 8 or 9, further comprising sensing a first temperature with a first temperature sensor, the first temperature sensor in communication with a power gate, the power gate throttling power to the first memory if a first temperature threshold is reached.

11. 11. The method of claim 10, further comprising sensing a second temperature with a second temperature sensor, the second temperature sensor in communication with the power gate, and the power gate throttling power to the first memory if a second temperature threshold is reached.

12. further comprising communicating between an error correction engine, the first memory, and the processor; The method of any one of claims 8 to 11, wherein data bits and check bits for the error correction engine are co-located.

13. 1. A computer program for memory storage and retrieval, said computer program comprising: fetching a first task from a memory die of a non-volatile memory via a vertical interconnect using a first buffer segment of a first memory, the first memory having the first buffer segment and a second buffer segment; submitting the first task to a compute core via the first buffer segment; prefetching a second task from the memory die via the vertical interconnect using the second buffer segment while the compute core is processing the first task; submitting the second task to the compute core via the second buffer segment upon completion of processing the first task; and processing the second task by the compute core; A computer program for executing the above.

14. The computer program causes the computer to: generating a first task computation result in response to the compute core processing the first task; transmitting, by the processor, the first task calculation result to the first memory; receiving the first task calculation result by the first memory; and further executing sending the first task calculation result to the memory die; 14. The computer program product of claim 13, wherein the first memory is a lower density memory than the memory die.

15. A computer-readable recording medium for recording the computer program according to claim 13 or 14.

Images