On-package and off-package combined memory system

Abstract

The present disclosure relates to a packaged-on and off-chip combined memory system using a custom base layer in which one or more dedicated interfaces to off-chip memory are fabricated. Packaged-on processors and packaged-on memory are also directly coupled to the custom base layer. The custom base layer includes memory management logic between the processors and the memory (both packaged-on and off-chip) to direct requests. These memories are disclosed as a combined memory space with greater bandwidth and capacity than either off-chip memory or packaged-on memory alone. The memory management logic services requests while maintaining quality of service (QoS) to each allocated bandwidth requirement. The allocations can include any combination of packaged-on and / or off-chip memory. The memory management logic also manages data migration between the packaged-on memory and the off-chip memory.

Description

BACKGROUND

[0001] Recently, high bandwidth memory access has been achieved by coupling a processor to a stack of memory dies and buffers through an interposer. Wires fabricated in the interposer transmit signals between the processor and the buffers stacked with the memory dies. The interposer provides increased density and speed compared to traditional printed traces or wire bond connections, enabling high speed parallel access to more bits of memory. However, even when stacked memory is coupled to a processor using an interposer, artificial intelligence, machine learning, and other compute intensive applications can still be limited by memory bandwidth and also require more memory capacity than can be provided by a stack of memory dies. There is a need to address these issues and / or other issues associated with the prior art. SUMMARY

[0002] Embodiments of the present disclosure relate to a packaged on and off combined memory system. A custom base layer (including one or more dedicated interfaces to packaged off memory) is directly coupled to a packaged on processor and packaged on memory. A single device package encloses the processor, custom base layer, package substrate, and at least one memory stack. The custom base layer includes memory management logic between the processor and memory (packaged off and packaged on) to direct requests. These memories are exposed as a combined memory space with greater bandwidth and capacity compared to separate packaged off memory or packaged on memory. The memory management logic services memory access requests to provide bandwidth requirements for each processing workload while maintaining quality of service (QoS). The memory can be organized into different pools, which include any combination of packaged on and / or packaged off memory, to meet different bandwidth requirements. The memory management logic also manages data migration between different pools.

[0003] Compared to traditional systems such as those described above, the memory management logic exposes the combined memory space to the processor, which provides a unified interface for accessing more memory capacity at greater bandwidth and speed. The memory management logic can also offload address translation and mapping operations from the processor. The packaged on and off combined memory system provides high memory bandwidth for artificial intelligence, machine learning, and other compute intensive applications and also provides greater memory capacity than can be provided by stacked memory dies alone.

[0004] Methods and systems for on-package and off-package combined memory systems are disclosed. In one embodiment, a memory access request is received from a processor coupled to a base layer within a package, and the memory access request is processed by memory management logic fabricated within the base layer (coupled between the processor and the memory system). The memory management logic can send a first portion of the memory access request to at least one memory stack enclosed within the package and directly coupled to the base layer, and send a second portion of the memory access request to at least one memory device located outside the package and coupled to the package through a first connection. The memory access requests in the first portion and the second portion can be interleaved.

[0005] In one embodiment, a processor is coupled to a base layer within a package, the package encloses the processor and the base layer, and the base layer includes memory management logic and is coupled between the processor and a memory system. The memory management logic processes memory access requests received from the processor. The memory system includes at least one memory stack enclosed within the package and directly coupled to the base layer, and at least one memory device located outside the package and coupled to the package. BRIEF DESCRIPTION OF DRAWINGS

[0006] Systems and methods for on-package and off-package combined memory systems are described in detail below with reference to the accompanying drawings, wherein:

[0007] Figure 1A A block diagram of an example processing system including a processor coupled to an on-package and off-package combined memory system suitable for implementing some embodiments of the present disclosure is shown.

[0008] Figure 1B A flow diagram of a method for processing memory access requests for an on-package and off-package combined memory system according to one embodiment is shown.

[0009] Figure 2A A block diagram of an example logical base layer suitable for implementing some embodiments of the present disclosure is shown.

[0010] Figure 2B A computing system including an on-package and off-package combined memory system according to one embodiment is shown.

[0011] Figure 3A A conceptual diagram of near and far memory mapping suitable for implementing some embodiments of the present disclosure is shown.

[0012] Figure 3B A flow diagram of a method of data migration suitable for implementing some embodiments of the present disclosure is shown.

[0013] Figure 4An example parallel processing unit suitable for implementing some embodiments of the present disclosure is shown.

[0014] Figure 5A A conceptual diagram of a processing system implemented using a PPU suitable for implementing some embodiments of the present disclosure. Figure 4

[0015] Figure 5B An example system in which various previous embodiments can be implemented is illustrated.

[0016] Figure 5C Components of an example system that can be used for training and utilizing machine learning in at least one embodiment are illustrated.

[0017] Figure 6 An example streaming system suitable for implementing some embodiments of the present disclosure is illustrated. DETAILED DESCRIPTION

[0018] Methods and systems relating to on-package and off-package combined memory systems are disclosed that enable capacity-optimized and bandwidth-optimized dynamic random access memory (DRAM) devices to be used in a synergistic manner to achieve higher capacity and higher bandwidth to a processor compared to traditional solutions. This allows legacy workloads that are only bandwidth sensitive to continue using high bandwidth HBM, but also allows newer capacity-limited workloads to be deployed on the processor and can benefit from increased memory bandwidth compared to traditional solutions. When the combination of HBM and at least a portion of the off-package memory system provides a unified memory pool compared to a memory pool that includes only HBM, bandwidth-sensitive workloads can also benefit from increased memory bandwidth.

[0019] The traditional assumption of compute directed by capacity of graphics and high performance computing (HPC) workloads is being transformed by data center applications. Emerging data center applications require higher memory capacity from processing systems compared to the memory capacity typically provided by scaling high bandwidth memory (HBM) die density, number of layers, or number of HBM sites in a single package. More specifically, recent applications such as recommendation systems and extract, transform, and load (ETL) operations that prepare data for performing data science and analytics pipelines require fast access to terabytes of memory capacity.

[0020] ​In addition to scaling the capacity of HBM, conventional techniques are used to provide a greater physical address space for a processor to access peer or host memory through an interface with lower bandwidth than the local HBM. Recently, processors include a serial interface that connects to an off-chip buffer coupled with a double data rate (DDR) dual in-line memory module (DIMM) that essentially provides an extension of the edge of the die to enable high capacity. The processor interface to the off-chip buffer is separate from the interface with the local HBM, such that the processor bears the responsibility of maintaining information for sending requests to the correct interface.

[0021] Figure 1A A block diagram of an example processing system 100 is shown that includes a processor 110 coupled to an on-package and off-package combined memory system suitable for implementing some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete components or distributed across many components, or combined with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by an entity can be performed by hardware, firmware, and / or software. For instance, various functions can be performed by a processor executing instructions stored in memory. Also, those of ordinary skill in the art will recognize that any system that executes the operations of a processor 110 coupled to an on-package and off-package combined memory system is within the scope and spirit of embodiments of the present disclosure.

[0022] In one embodiment, the processor 110 is a graphics processing unit (GPU). The processor 110 is coupled to the on-package DRAM stacks 115 and the off-package memory modules 145 through an interposer 125. In one embodiment, each DRAM stack 115 includes a memory die stacked on top of an HBM controller 120 that is coupled vertically to each die in the DRAM stack 115 to form an HBM stack. One or more HBM stacks are enclosed with the processor 110 in a package 105. In one embodiment, each of the DRAM stacks 115 is coupled to a respective HBM controller 120 through a parallel DRAM bus. In one embodiment, at least one HBM stack is contained in the package 105.

[0023] Compared to the package-on-HBM stacks enclosed within package 105, memory modules 145 are located outside of package 105. In one embodiment, each package-external memory module 145 includes at least one low-power (LPDDR) or DDR memory device, die, or component. In one embodiment, each memory module 145 includes one buffer (e.g., a high-speed serial-to-LP5 buffer or interface) coupled between bus or connection 112 or 114 and the DRAM memory components. In one embodiment, the buffer within each memory module 145 is coupled to one or more DRAM memory components through a parallel bus. In one embodiment, each buffer within memory module 145 is coupled to at least one memory stack, e.g., four 4GB stacks, through a parallel DRAM bus, for a total package-external memory capacity of 16GB within each memory module 145. In one embodiment, memory modules 145 include DIMMs. In one embodiment, connections 112 and 114 are high-speed serial links, e.g., JEDEC DDR or LPDDR connections to memory modules 145. In one embodiment, the high-speed serial link provided by each of connections 112 and 114 provides a data transfer rate or bandwidth of 128 GB / s in each direction.

[0024] Processor 110 is directly coupled to logic base layer 130 through connection 106, which is capable of supporting greater bandwidth than that provided by package-external memory modules 145 or HBM stacks. In one embodiment, processor connection 106 is capable of supporting the sum of the bandwidth provided by package-external memory modules 145 and HBM stacks. More specifically, within package 105, short reach interposer signaling can be used between processor 110 and logic base layer 130. In one embodiment, logic base layer 130 is a custom silicon layer that includes fabricated logic. Interposer 125 provides conductive connections for the transmission of signals between processor 110 and logic base layer 130, as well as between each memory module 145 and logic base layer 130. Signals transmitted between processor 110 and logic base layer 130 do not exit package 105 and can be transmitted through conductive paths fabricated in base layer 130. Similarly, connections 102 and 104 between package-on-HBM controller 120 and logic can also be transmitted through conductive paths fabricated in base layer 130. As shown, HBM controller 120 is directly coupled to logic base layer 130 through connections 102 and 104. In one embodiment, connections 102 and 104 are each a parallel DRAM bus. Figure 1A

[0025] ​The interposer 125 is located on the top surface of the package substrate 135, and the opposite (bottom) surface of the package substrate 135 is physically and electrically coupled to the external substrate 140. In one embodiment, the package 105 is a ball grid array, and electrical signals are transmitted through conductive solder balls located between the package substrate 135 and the external substrate 140. The dedicated connections 112 and 114 between each of the package off memory modules 145 and the logic base layer 130 transmit signals through conductive paths fabricated in the interposer 125, the package substrate 135, and the external substrate 140. In one embodiment, the external substrate 140 is a printed circuit board and the conductive paths are traces. In one embodiment, the memory modules 145 are each packaged in a ball grid array mounted on the external substrate 140. In one embodiment, the conductive paths for the connections 112 and 114 within the interposer 125 and the package substrate include through-silicon vias. The dedicated connections 112 and 114 couple the memory modules 145 directly to the logic base layer 130.

[0026] The logic base layer 130 aggregates the memory capacity provided by both the DRAM stack 115 and the memory modules 145, freeing the processor 110 from most, if not all, memory mapping and management operations. Logic circuitry fabricated in the logic base layer 130 performs the aggregation, memory mapping, and management operations. The logic base layer 130 includes separate interfaces for each of the on-package processor 110, the on-package HBM stack, and the package off memory modules 145. In one embodiment, each HBM stack has a 24 GB capacity and provides 512 GB / s of memory access bandwidth (any combination of read and write access). In one embodiment, each memory module 145 provides 64 GB of additional package off memory capacity, and each of the connections 112 and 114 provides an additional 128 GB / s of memory access bandwidth to the processor 110. Assuming a configuration with a single DRAM stack 115 and two memory modules 145, the total memory capacity is 152 GB, and 768 GB / s of memory bandwidth is provided to the processor 110 through the aggregation of memory bandwidth by the logic base layer 130 for the combination of on-package and package off memory.

[0027] In general, the available bandwidth of on-package memory is greater than that of package off memory, and the available capacity of on-package memory is less than that of package off memory. The logic base layer 130 enables support for both on-package memory and package off memory, providing the high memory bandwidth required by traditional applications while also meeting the high memory capacity requirements of data center and machine learning applications.

[0028] The base layer provides a combination of on-package memory and off-package memory as compared to traditional central processing unit (CPU) solutions that only provide off-package memory implementations. The logical base layer 130 provides a single unified memory with elevated bandwidth as compared to traditional GPU solutions that provide on-package memory and rely on off-package low bandwidth host memory for additional capacity. In one embodiment, the logical base layer 130 includes memory management logic required by the processor 110 for transparent memory space access, which includes providing configurable memory space to meet bandwidth requirements and capacity requirements of different workloads.

[0029] More illustrative information will now be set forth in relation to various optional architectures and features that can be utilized in implementing the aforementioned framework in accordance with a user's desires. It should be strongly noted that the following information is set forth in relation to the exemplary embodiments, and that modifications to the embodiments can be made without departing from the spirit and scope of the disclosure. Any feature(s) described in relation to any of the embodiments can be selectively incorporated into any other embodiment.

[0030] Figure 1B A flow diagram illustrating a method 150 for processing memory access requests for a combined on-package and off-package memory system is shown in accordance with one embodiment. Each block of the method 150 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on computer storage media. The method can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name but a few. Further, the method 150 is described by way of example in relation to a processor 110 coupled to a combined on-package and off-package memory system Figure 1A However, such a method can additionally or alternatively be performed by any one system or combination of systems, including but not limited to those described herein. Moreover, those of ordinary skill in the art will understand that any system performing the method 150 is within the scope and spirit of the embodiments of the disclosure.

[0031] In step 155, a memory access request is received from a processor, the processor directly coupled to a base layer within a package, and the memory access request is processed by memory management logic fabricated in the base layer coupled between the processor and the memory system. In one embodiment, the base layer is the logical base layer 130, and the processor is the processor 110. In one embodiment, the processor 110 is coupled to a memory system that includes the logical base layer 130, at least one HBM stack, and at least one off-package memory device (e.g., the memory module 145). In one embodiment, the capacity of the at least one memory stack is less than the capacity of the at least one memory device. In one embodiment, the data transfer rate of the at least one memory stack is greater than the data transfer rate of the at least one memory device. In one embodiment, the logical base layer 130 includes the memory management logic.

[0032] In step 160, the memory management logic sends a first portion of the memory access request to the at least one memory stack, the stack enclosed in a package and directly coupled to the base layer. In one embodiment, in response to the first memory access request, data stored in the at least one memory stack and the at least one memory device is transferred between the processor and the base layer over the connection 106. In one embodiment, the memory management logic translates each address associated with the memory access request according to an interface map to send the memory access request to one of the at least one memory stack or the at least one memory device. In one embodiment, the interface map is stored in the logical base layer 130 and maps each address associated with the access request to one of the interfaces coupled with the connections 102, 104, 112, and 114. In one embodiment, the logical base layer 130 translates the address to identify one of the interfaces and transfers the address or at least a portion of the translated address on the identified interface.

[0033] In step 165, the memory management logic sends a second portion of the memory access request to the at least one memory device, the memory device located outside the package and coupled to the base layer through the first interface. In one embodiment, the at least one memory device is coupled to the base layer through the first interface and a memory system, and the memory system further includes at least one additional memory device, the at least one additional memory device located outside the package and coupled to the base layer through a second interface separate from the first interface. In one embodiment, the remote memory pool or space includes only locations in the at least one memory device. In one embodiment, the near memory pool includes a first location in the at least one memory stack and a second location in the at least one memory device. In one embodiment, the first location and the second location are interleaved within the near memory pool according to an address mapping performed by the memory management logic.

[0034] In one embodiment, data is simultaneously transferred between the memory management logic and at least one memory stack and at least one memory device. For example, in response to a memory read request, data is simultaneously sent from at least one of the DRAM stacks 115 and at least one of the memory modules 145 to the logic base layer 130. In another embodiment, in response to a memory write request, data is simultaneously sent from the logic base layer 130 to at least one of the DRAM stacks 115 and at least one of the memory modules 145. In one embodiment, each memory access request to the DRAM stacks 115 and the memory modules 145 is handled by the logic base layer 130.

[0035] Figure 2A A block diagram of an example logic base layer 130 suitable for implementing some embodiments of the present disclosure is shown. The logic base layer 130 includes an HBM controller 220 that is connected through an interface to connections 102 and 104 for on-package memory, which are coupled to the logic base layer 130 at HBM site 235. The logic base layer 130 also includes memory device controllers 212 and 214 that are connected through an interface to connections 112 and 114, respectively, which are connected to off-package memory including memory modules 145. The logic base layer 130 also includes a processor interface 215 that directly couples the logic base layer 130 to the processor 110 through connection 106. In one embodiment, the logic base layer 130 includes one or more additional processor interfaces 215 that are coupled to one or more additional processors, which can be the same as or different from the processor 110. In one embodiment, the logic base layer 130 includes at least one additional interface that is coupled to one or more other devices that access the combined on- and off-package memory. Any processor or device that accesses the on- and off-package memory issues requests to the logic base layer 130, as there is no alternate path to / from the HBM stacks or off-package memory.

[0036] The memory management logic 225 directs memory access requests received from the processor 110 to the correct memory via the processor interface 215 according to address mapping information that can be stored in the page table 240. This address mapping information can include interface mappings and be used to translate addresses received from the processor 110 to interface-specific addresses. In one embodiment, interface mapping information can also be stored in the page table 240 or can be stored separately from the address mapping information. Thus, the processor 110 need not be aware of the different memories and can be relieved of performing memory translations for on- and off-package memory. The memory management logic 225 also responds to requests received from the processor 110 via the processor interface 215.

[0037] Memory management logic 225 routes each request for an address that maps to an off-package memory module 145 coupled to memory device controller 212 through connection 112. Similarly, memory management logic 225 routes each request for an address that maps to an off-package memory module 145 coupled to memory device controller 214 through connection 114. In one embodiment, memory device controllers 212 and 214 translate addresses received from memory management logic 225 as necessary to access locations in one or more memory modules 145. In one embodiment, address mapping information specific to interfaces 112 and 114 is stored in memory device controllers 212 and 214. Memory device controllers 212 and 214 can also implement any communication protocol specified by memory modules 145. In one embodiment, memory device controllers 212 and 214 are not fabricated in logic base layer 130, but instead are located off-package and integrated into each of memory modules 145, or between package 105 and respective memory modules 145.

[0038] Memory management logic 225 routes each request for an address that maps to an on-package HBM stack coupled to interfaces 102 and 104 through HBM controller 220. In one embodiment, HBM controller 220 translates addresses received from memory management logic 225 as necessary to access locations in the HBM stack. HBM controller 220 can also implement any communication protocol specified by the HBM stack.

[0039] To implement various different memory system configurations, the base layer can be redesigned to support different memory devices and / or additional interfaces can be removed or added without redesigning or modifying processor 110. Furthermore, by allocating a memory pool for legacy applications at locations in DRAM stack 115, legacy applications that rely on bandwidth provided by the HBM stack can be supported without modifying the legacy applications. Bandwidth-limited applications can benefit from some increased memory capacity that can use a near memory pool allocated bandwidth of all DRAM stack 115 and all off-package memory, resulting in, for example, 50% more bandwidth and 50% more capacity than the HBM stack alone. Such an allocation will not guarantee available bandwidth of the far memory pool, and much of the off-package memory capacity will not be utilized, but can achieve optimal performance for some "legacy" applications. Applications that can benefit from higher memory capacity available only using the on-package HBM stack can then use a memory pool allocated locations in off-package memory or a combination of off-package memory and on-package HBM stack to access the greater capacity.

[0040] The combined memory pool provided by the on-package and off-package memory is exposed in the processor 110 as a unified memory space. The combined memory pool can be divided into a large capacity, far memory (lower bandwidth) pool and a small capacity, near memory (higher bandwidth) pool. In one embodiment, the application program or system software is aware of the two memory pools and is responsible for paging in-and-out of data in the near memory and / or appropriately partitioning the data. In another embodiment, the management of the near and far memory is handled by the memory management logic 225 and is transparent to the application program and the processor 110.

[0041] In one embodiment, the near memory pool includes only the on-package HBM stack and the far memory pool includes only the off-package memory. In one embodiment, the near memory pool includes the on-package HBM stack and at least one "carve out" portion of the off-package memory. In one embodiment, the far memory pool includes the off-package memory not included in the near memory pool. In one embodiment, the address mapping for memory allocation in the near memory pool can be interleaved in the on-package / off-package memory to meet capacity and bandwidth requirements, as described in connection with FIG. 2. Figure 3A Processing workloads not limited by memory capacity can benefit from the additional bandwidth provided by the off-package memory in the interleaved mode. Capacity intensive workloads can benefit from the additional capacity provided by the off-package memory. The off-package memory accessed directly by the processor 110 through the logical base layer 130 has higher bandwidth compared to traditional solutions that use peer or host memory to provide additional capacity.

[0042] In an embodiment, a migration engine is included in the logical base layer 130 to move data or replicate data between different locations in the memory system in a manner transparent to the processor 110. In one embodiment, the migration of data between the near memory and the far memory does not increase the communication on the processor interface 215. Thus, the data migration is efficient in terms of power consumption and processor memory bandwidth. In one embodiment, the migration engine is implemented through the memory management logic 225. Various strategies can be used to perform the migration. For example, frequently accessed data can be migrated from the far memory to the near memory through the cache 230 without interrupting request servicing and / or causing slowdowns associated with regular management events, such as the flushing of translation lookaside buffers (TLBs). In one embodiment, the cache 230 stores data that maps to addresses only in the far memory.

[0043] The migration process can be triggered by an explicit request generated by the processor 110 as a result of program instructions, or can be triggered by a different mechanism. In one embodiment, migration can be triggered according to access frequency. In one embodiment, migration is triggered according to access frequency within a given time window. In one embodiment, migration is triggered based on access pattern, for example migrating pages that are located between two pages that have already been migrated. In one embodiment, migration is triggered based on analysis of addresses in the access stream to identify pages to migrate.

[0044] In one embodiment, data transfer between the near memory and the far memory can be done at a programmed or fixed granularity, such as a page size. In one embodiment, each cache line in the cache 230 stores data for a page of memory, and each location in the page is stored in a separate sector within the cache line. For example, in one embodiment, a sector is 32 bytes of data, and a cache line can store 2048 sectors. The memory management logic 225 can track the number of valid sectors in a cache line, and when the number of sectors is greater than a threshold, the memory management logic 225 can migrate the page associated with the cache line to the near memory. The data stored in the valid sectors is also stored to the corresponding memory location.

[0045] During the migration process, the memory management logic 225 can track the locations in the page that are being migrated. When a location is migrated to an invalid sector in the cache line, the page table is updated, and any requests for the valid sectors are serviced by the cache 230, while requests for invalid sectors that have not been migrated are serviced by loading the data from the far memory to the corresponding invalid sector. The memory management logic 225 can continue to process requests received from the processor 110 while the migration is in progress, even when the request accesses a location that is being migrated.

[0046] In one embodiment, the total memory capacity is the sum of the on-package and off-package memory, referred to as the exclusive mode. In contrast, for the inclusive mode, the far memory acts as a backing store for the near memory. The total memory capacity for the inclusive mode is the capacity of the far memory.

[0047] Cache 230 can be used to perform the migration. For the inclusive mode, when a memory page triggers a migration, memory management logic 225 loads the invalid sectors into the cache line allocated to the memory page while continuing to service new requests. If the memory page does not exist in the cache, memory management logic 225 first allocates a cache line to the memory page and then loads the sectors. In one embodiment, when the invalid sectors are loaded, the cache line is locked in the cache and when the entire cache line is valid (e.g., cache behavior "full"), memory management logic 225 updates page table 240 accordingly. For example, the location of the migrated page can be changed to the location of the near memory and the cache line can be marked as "dirty." In one embodiment, memory management logic 225 stores interface information used to determine routing requests and updates the interface information to associate the address of the migrated page with the near memory interface. Once the migration is complete, memory management logic 225 can unlock the cache line and, if necessary, evict the cache line.

[0048] For the exclusive mode, all near memory pools and far memory pools can be used and cache 230 can be used to "swap" data between the near and far memory pools. A first page of data migrated from the far memory to the near memory can be temporarily stored in cache 230 while the location in the far memory storing the first page is overwritten by a second page of data migrated from the near memory to the far memory. As previously described, the first page can be associated with a cache line and memory management logic 225 can load the invalid sectors into the cache line while continuing to service new requests. Once the entire cache line is valid, the first page of data can then be stored to the second location in the near memory and memory management logic 225 updates page table 240 to indicate the second page in the far memory and the first page in the near memory. In essence, by using cache 230 as temporary storage, the first page is "swapped" with the second page. In another embodiment, the second page of data can be stored to cache 230 and the first page of data can be stored directly to the second location in the near memory. The second page of data can then be read from cache 230 and stored in the far memory and memory management logic 225 can update page table 240 accordingly.

[0049] When unused space is available in the near memory, a first page of data can be stored directly to a second location in the near memory without first being stored in the cache 230. Once the first page of data is stored in the near memory, the memory management logic 225 updates the page table 240 accordingly. In one embodiment, when the near memory includes at least a portion of the off-package memory, a portion of the data in the off-package portion of the near memory is not migrated to the far memory by moving the portion. Rather, the allocation of the location of the portion of the data stored in the off-package memory is changed from the near memory to the far memory. Thus, only the remaining data stored in the on-package memory needs to be moved to the far memory to complete the migration of the near memory to the far memory.

[0050] In the context described below, a page of data can exist in one of three states (resident in on-package memory, resident in off-package memory, or in migration). When the memory management logic 225 determines that a page is in migration, the current location of the data in the cache line that stores the page in relation to the particular sector is also determined. The request can be completed when the sector is valid. Additionally, the request can be completed once the sector becomes valid. In one embodiment, the memory management logic 225 suspends processing of the request until the migration is complete.

[0051] Figure 2B A computing system 200 including an on-package and off-package combined memory system 255 is shown in accordance with one embodiment. The computing system 200 includes a processor 250, a logical base layer 130, and a memory system 255. The memory system 255 includes on- and off-package memory, such as at least one HBM stack (e.g., HBM controller 120 and DRAM stack 115) and at least one memory module 145, respectively.

[0052] In one embodiment, processor 250 is coupled to multiple logical base layers 130, processor 250 includes a separate interface 106 to each logical base layer 130. The multiple logical base layers 130 can be fabricated in a single die or multiple dies and coupled to separate memory systems 255. In one embodiment, links between the logical base layers 130 are fabricated within the interposer 125. Processor 250 routes requests to the correct interface 106 and logical base layer 130 based on the address mapped by the interface. In one embodiment, each logical base layer 130 is coupled to a single DRAM stack 115 and two memory modules 145, providing 152 GB of total memory capacity and 768 GB / s of memory bandwidth. When six logical base layers 130 and six memory systems 255 are included in the computing system 200, a total of 912 GB of memory capacity can be provided by the combination of on-package and off-package memory, and a bandwidth of 4.5 terabytes per second can be provided to processor 110 by the aggregation of the memory bandwidth of the six logical base layers 130 of the combination of on-package and off-package memory.

[0053] Processor 250 can be processor 110 and include an interface 280 coupled to at least one additional device that is external to the package, the package at least encompassing processor 250, logical base layer 130, and the on-package portion of memory system 255. In one embodiment, the at least one additional device is additional memory that can be accessed by processor 250. In a conventional system, the at least one additional device is used to increase memory capacity when only on-package HBM stacks are available. However, the at least one additional device is not coupled to processor 250 through the same interface as the on-package HBM stacks compared to the off-package memory included in memory system 255.

[0054] Processor 250 can also include one or more units including storage, such as L2 cache 260 and Ll cache 265, crossbar 270 interconnect, and / or processing or execution core 275. In one embodiment, processor 250 can include multiple L2 caches 260. Each L2 cache 260 can be individually connected to a dedicated logical base layer 130. When multiple memory systems 255 are included in computing system 200, processor 250 maintains mapping information to determine which of the memory systems 255 each request is sent to. In one embodiment, processor 250 can designate whether data is stored in near memory or far memory, but processor 250 does not need to track or control the mapping of addresses in a particular memory system 255. Based on the need for data storage, logical base layer 130 provides and maintains mapping information for data stored in memory system 255.

[0055] While data stored in memory system 255 can be cached in L2 cache 260 and / or LI cache 265, migration of data (between the far memory pool and the near memory pool) in each of memory system 255 is performed by logical infrastructure layer 130 without the need to transfer data between logical infrastructure layer 130 and processor 250. In comparison to using only on-package memory, logical infrastructure layer 130 and memory system 255 provide processor 250 with higher memory capacity and higher memory bandwidth.

[0056] In one embodiment, at least a portion of processor 250 is included in a virtual machine. In one embodiment, processor 250 is part of a server or data center, and data stored in memory system is streamed to a user device. In one embodiment, processor 250 generates data for training, testing, or proving a neural network used in a machine, robot, or autonomous vehicle. In one embodiment, at least a portion of processor 250 is included in a cloud computing environment.

[0057] Figure 3A Conceptual diagrams illustrating near memory and far memory mapping suitable for implementing some embodiments of the present disclosure are shown. In one embodiment, on-package memory and off-package memory contained in memory system 255 are divided into near memory 310 and far memory 320. In one embodiment, the entirety of near memory 310 includes all on-package memory and excludes off-package memory. In one embodiment, the entirety of far memory 320 includes all off-package memory and excludes on-package memory. When near memory 310 is only on-package memory and far memory 320 is only off-package memory, the memory bandwidth provided by each memory pool is easily determined. Applications and / or processing workloads can be allocated memory in different pools based on memory bandwidth requirements and / or memory capacity requirements. In each pool, addresses can be mapped (e.g., striped) across different memory modules 145 and HBM stacks at a cache line or other quantity of granularity.

[0058] In one embodiment, near memory 310 includes all on-package memory and a portion of off-package memory, while the remaining off-package memory is included in far memory 320. In one embodiment, interleaved mapping is used for near memory 310 such that within an address range (linear or interleaved), locations in off-package memory are interleaved with locations in on-package memory. In one embodiment, the interleaving granularity is 64K bytes, such that Figure 3AThe illustrated interleaving region includes 7 interleaved 64K byte memory portions. Interleaving ensures that when a range of addresses is accessed, the memory bandwidth can be higher compared to accessing only the on-package memory. In one embodiment, the interleaving granularity is a power of 2, 2 n where n is an integer. In one embodiment, the interleaving granularity is finer than 64K bytes, such as a value of 512B-1K bytes. In another embodiment, the interleaving granularity is coarser than 64K bytes.

[0059] The memory bandwidth can be higher because the logic base layer 130 enables communication between the processor 110 or 250 and both the on-package memory and the off-package memory to occur simultaneously. Thus, interleaving of off-package memory locations with on-package memory locations increases the bandwidth up to the point of off-package memory bandwidth saturation between the memory system 255 and the logic base layer 130 or off-package memory bandwidth saturation between the logic base layer 130 and the processor 250. In one embodiment, the ratio of off-package memory channels to on-package memory within the near memory is equal to the ratio of the aggregate bandwidth of the two different types of memory (HBM stack and off-package memory).

[0060] For example, when three memory modules 145 are included in the memory system, each having a separate 128GB / s interface (e.g., channel), the bandwidth of the off-package memory is 384GB / s. If the bandwidth of the on-package memory is 512GB / s, the aggregate bandwidth is 896GB / s, with the off-package memory providing 3 / 7 of the aggregate bandwidth and the on-package memory providing 4 / 7 of the aggregate bandwidth. As Figure 3A As illustrated, the interleaving of 3 off-package portions to 4 on-package portions within a region is repeated for the additional regions within the mapping of the near memory 310. Depending on the bandwidth ratio, if 16GB of on-package memory is included in the near memory, 12GB of off-package memory is allocated to the near memory, for a total near memory capacity of 28GB with a bandwidth of 896GB / second. The remainder of the off-package memory is allocated to the far memory, and the addresses are striped across three memory modules 145, providing the remaining memory capacity shared with the near memory at a bandwidth of up to 384GB / s.

[0061] In another example, each pool is allocated a maximum bandwidth, and the near memory uses the maximum bandwidth to receive all of the on-package bandwidth and a portion of the off-package bandwidth. Specifically, the near memory can be allocated 2 / 3 of the total off-package memory bandwidth of 384 GB / s (i.e., 256 GB / s) of off-package bandwidth. In total, the aggregate bandwidth of the near memory is limited by the maximum bandwidth of the interface 215 between the logical substrate 130 and the processor 110. However, the maximum bandwidth of the interface 215 can be greater than the total bandwidth of the memory system, such that the aggregate bandwidth is instead limited by the total bandwidth of the memory system. In one embodiment, the processor bandwidth provided by the interface 215 is almost twice the on-package memory bandwidth. When the near memory is allocated 2 / 3 of the off-package bandwidth, the maximum bandwidth of the near memory is 768 GB / s, the near memory is allocated 8 GB of off-package memory, and when combined with 16 GB of on-package memory, the total near memory capacity is 24 GB. On average, the amount of time taken to access all 8 GB of off-package memory is the same as the amount of time taken to access 16 GB of on-package memory. In this example, when the near memory bandwidth of 768 GB / s is maximized, only 256 GB / s of off-package bandwidth is used. This leaves 128 gb / s of reserved bandwidth to request the far memory pool. Allocating more off-package memory to increase the near memory capacity and bandwidth would reduce the reserved bandwidth available for far memory requests.

[0062] In one embodiment, the logical substrate 130 prioritizes near memory accesses over far memory accesses to prevent far memory requests from reducing the maximum bandwidth required to include the portion of off-package memory in the near memory pool. In one embodiment, accesses are arbitrated according to the maximum bandwidths associated with the near memory pool and the far memory pool, and near memory accesses can temporarily consume more than their associated maximum bandwidth when there are no far memory accesses. Similarly, far memory accesses can temporarily consume more than their associated maximum bandwidth when there are no near memory accesses. In one embodiment, a threshold is used to identify underutilization of the near memory such that the far memory can temporarily consume more than its maximum bandwidth while near memory accesses are prioritized over far memory accesses.

[0063] While interleaving balances the memory bandwidth across all memory in different memory pools (near memory pool and far memory pool), it also increases the overall access latency because accessing off-package memory has a larger latency than accessing on-package memory. When using interleaving, the off-package memory bandwidth is effectively divided between the near memory and far memory pools such that far memory traffic does not reduce the bandwidth available for near memory requests. In one embodiment, the logic substrate 130 determines the interleaving ratio based on the memory bandwidth of different memory types and arbitration between near memory and far memory accesses as needed to ensure that the amount of memory bandwidth is provided and QoS is maintained.

[0064] Applications that cannot benefit from greater memory capacity will realize an increase in memory bandwidth from the additional bandwidth of off-package memory in the interleaving mode. Workloads that are limited in memory capacity can access off-package memory at higher bandwidths compared to traditional solutions that rely on slower interfaces (in terms of bandwidth and / or latency) to access additional memory.

[0065] Figure 3B A flow diagram of a method 350 for data migration suitable for implementing some embodiments of the present disclosure is shown. Each block of the method 350 described herein comprises a computational procedure that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be carried out by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage media. The method can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Figure 1A The processor 110 of a package on and off combined memory system coupled to Figure 2B The processor 250 of a memory system 255 coupled to the logic substrate 130. However, the method can additionally or alternatively be executed by any one system or any combination of systems, including but not limited to those described herein. Moreover, one of ordinary skill in the art would understand that any system executing the method 350 is within the scope and spirit of embodiments of the present disclosure.

[0066] In step 355, a memory access request for data stored in a remote memory pool is received. In one embodiment, the memory access request is received from a processor that is directly coupled to a base layer in a single package with the processor, and the base layer is coupled between the processor and a memory system. The memory system includes on-package memory and off-package memory, where the on-package memory is enclosed in the single package and the off-package memory is located outside of the single package. In one embodiment, the near memory pool includes a combination of the on-package memory and the off-package memory, and the remote memory pool includes only the off-package memory.

[0067] In step 360, memory management logic fabricated in the base layer determines whether a migration is triggered. In one embodiment, a migration is triggered when a threshold number of accesses associated with the access request is exceeded. For example, an access request specifies an address in a page of memory, and the number of accesses to the page within a time period is tracked by the memory management logic for comparison to a threshold. When an access request does not result in a cache hit, data can be loaded into the cache from a location in the remote memory pool and initialize the number of accesses. If an empty cache line is not available, data stored in a cache line can be evicted. Based on information in a page table, any dirty sectors in the evicted cache line are written back to a location in the near memory or the remote memory. In an embodiment, the memory management logic determines that a threshold number of sectors have been loaded into cache lines of a page to trigger a migration process for the page.

[0068] If a migration is not triggered in step 360, the access request is processed in step 370. Otherwise, in step 365, the memory management logic migrates lost sectors from the remote memory pool to the near memory pool. In one embodiment, a cache line stores a plurality of sectors of a memory page, and sectors storing the page data are valid, and sectors not storing the page data are invalid and considered lost. In one embodiment, the migration process started in step 365 can be triggered or initiated by executing an instruction.

[0069] The migration process can be performed without slowing the processing of access requests. In one embodiment, processor initiated requests are processed unconditionally and the page migration is performed in the background as appropriate without interfering with the processing of processor initiated requests. Thus, the migration process is initiated in step 365 and then the access requests are processed in step 370 while the page migration continues in step 375. For example, the first 512 sectors of a page are accessed, triggering the migration of the page, and the first 100 sectors are migrated before the processor requests sectors 200-300. The request for sectors 200-300 is serviced (and loaded into the cache) before jumping to sector 301 and then the migration resumes copying sectors 101-199 to continue the migration. Because the processor initiated accesses complete some of the migration before the migration engine, no redundant work is performed for that migration. The page migration is transparent in terms of processing processor initiated accesses (no TLB flushes, etc.) and in terms of access performance.

[0070] In step 375, the memory management logic determines whether the migration of the entire page is complete and, if not, the memory management logic returns to step 365 to migrate the missing sectors. When the migration is complete, in step 380, the memory management logic updates the mapping information for the page to indicate that the page is included in the near memory pool.

[0071] In one embodiment, the near memory pool includes a first location in at least one memory stack and a second location in at least one memory device. The first and second locations can be interleaved in the near memory pool according to address mapping performed by the memory management logic. In one embodiment, in response to a trigger, the memory management logic migrates first data stored in a third location in the at least one memory device that is not included in the near memory pool to the near memory pool. The trigger can be that a threshold number of memory access requests to the third location are exceeded in a time period. In one embodiment, portions of the first data are copied from the third location to the near memory pool as at least one additional memory access request is processed. In one embodiment, the memory management logic updates an address mapping for the first data to indicate that the first data is stored in the near memory pool after the first data is migrated. In one embodiment, the memory management logic migrates data stored in the near memory pool to a fourth location in the at least one memory device that is not included in the near memory pool and updates an address mapping to indicate that the data is stored in the fourth location.

[0072] Providing a processor with unified memory including on-package and off-package memory through a base layer can increase both available memory capacity and memory bandwidth. The on-package memory type can be different compared to the off-package memory type with a corresponding interface provided through the base layer. The bandwidth between the processor and the base layer can be higher compared to the bandwidth of the on-package memory or the off-package memory. In one embodiment, the memory capacity is divided into near and far memory pools, and the portion of off-package memory allocated to the near memory pool is interleaved with the portion of on-package memory to provide higher aggregate memory bandwidth compared to using only on-package memory for the near memory pool.

[0073] Logic fabricated in the base layer offloads memory management tasks from the processor and enables various different memory configurations to have different interfaces, signaling techniques, capacities, device types, etc. Based on address and / or interface mapping information stored in the base layer, the logic can direct requests to the correct memory. The logic can also migrate data between the near and far memory pools even while continuing to process access requests. This logic decouples the processor from the memory system, enabling the processor and the memory system to evolve independently.

[0074] Using on- and off-package memory systems to increase available memory capacity can improve data center performance for workloads such as recommendation system training. When the processor 110 is a GPU, using on- and off-package memory systems can be beneficial for GPU-accelerated databases and other workloads where the working set is too large to fit into the on-package memory of a single GPU. Another service that can benefit from on- and off-package memory systems is virtualization of data centers, where different workloads with different memory bandwidth and capacity requirements can be served by a single processor using on- and off-package memory systems.

[0075] Parallel processing architecture

[0076] Figure 4A parallel processing unit (PPU) 400 according to one embodiment is shown. The PPU 400 can be used to implement the processor 110 within the processing system 100 or the processor 250 within the computing system 200. In one embodiment, a processor such as the PPU 400 can be configured to implement a neural network model. The neural network model can be implemented as software instructions executed by the processor, or in other embodiments, the processor can include a matrix of hardware elements configured to process a set of inputs (e.g., electrical signals representing values) to generate a set of outputs that can represent activations of the neural network model. In other embodiments, the neural network model can be implemented as a combination of processing by the matrix of hardware elements and software instructions. Implementing a neural network model can include determining a set of parameters for the neural network model through, for example, supervised or unsupervised training of the neural network model, and or alternatively, performing inference using the set of parameters to process a new set of inputs.

[0077] In one embodiment, the PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 400 is a latency hiding architecture designed to process many threads in parallel. A thread (e.g., an execution thread) is an instantiation of a set of instructions configured to be executed by the PPU 400. In one embodiment, the PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data in order to generate two-dimensional (2D) image data for display on a display device. In other embodiments, the PPU 400 can be used to perform general purpose computations. Although one exemplary parallel processor is provided herein for purposes of illustration, it is specifically intended that such processor be illustrative of any processor which can be substituted for the processor as appropriate and desirable for a specific application.

[0078] One or more PPUs 400 can be configured to accelerate thousands of high performance computing (HPC), data center, cloud computing, and machine learning applications. The PPU 400 can be configured to accelerate numerous deep learning systems and applications for autonomous vehicles, simulation, compute graphics such as ray tracing or path tracing, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0079] As Figure 4As shown, the PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a work distribution unit 425, a hub 430, a crossbar (Xbar) 470, one or more general processing clusters (GPCs) 450, and one or more memory partition units 480. The PPU 400 can be connected to a host processor and other PPUs 400 via one or more high-speed NVLinks 410. The PPU 400 can be connected to a host processor and other peripherals via an interconnect 402. The PPU 400 can also be connected to a local memory 404, which can include a number of memory devices. In one embodiment, the local memory can include a number of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem in which a number of DRAM dies are stacked.

[0080] The NVLink 410 interconnect enables systems to scale by connecting one or more CPUs with one or more PPUs 400, supports cache coherence between the PPU 400 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 410 to other units of the PPU 400 or from them, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In conjunction with the Xbar 470, the NVLink 410 allows inter-PPU communications. Figure 5B The NVLink 410 is described in further detail.

[0081] The I / O unit 405 is configured to transmit and receive communications (e.g., commands, data, etc.) from a host processor (not shown) over the interconnect 402. The I / O unit 405 can communicate directly with the host processor via the interconnect 402 or via one or more intermediate devices such as a memory bridge. In one embodiment, the I / O unit 405 can communicate with one or more other processors, such as one or more PPUs 400, via the interconnect 402. In one embodiment, the I / O unit 405 implements a Peripheral Component Interconnect Express (PCIe) interface for communications over a PCIe bus, and the interconnect 402 is a PCIe bus. In alternative embodiments, the I / O unit 405 can implement another type of known interface for communicating with external devices.

[0082] The I / O unit 405 decodes data packets received via the interconnect 402. In one embodiment, the data packets represent commands configured to cause the PPU 400 to perform various operations. The I / O unit 405 transmits the decoded commands to various other units of the PPU 400 that the commands can specify. For example, some commands can be transmitted to the front-end unit 415. Other commands can be transmitted to the hub 430 or other units such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, the I / O unit 405 is configured to route communications between and among various logical units of the PPU 400.

[0083] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload can include a number of instructions and data to be processed by those instructions. The buffer is a region of memory that is accessible (e.g., read / write) by both the host processor and the PPU 400. For example, the I / O unit 405 can be configured to access the buffer in system memory connected to the interconnect 402 via memory requests transmitted over the interconnect 402. In one embodiment, the host processor writes the command stream to the buffer and the PPU 400 transmits a pointer to the beginning of the command stream in response. The front-end unit 415 receives the pointer to the one or more command streams. The front-end unit 415 manages the one or more streams, reading commands from the streams and forwarding the commands to various units of the PPU 400.

[0084] The front-end unit 415 is coupled to a scheduler unit 420 that is configured to schedule various GPCs 450 to process tasks defined by one or more streams. The scheduler unit 420 is configured to track state information related to various tasks managed by the scheduler unit 420. The state can indicate which GPC 450 a task is assigned to, whether the task is active or inactive, a priority associated with the task, etc. The scheduler unit 420 manages execution of a plurality of tasks on the one or more GPCs 450.

[0085] The scheduler unit 420 is coupled to a work distribution unit 425, which is configured to distribute work to the GPCs 450 for execution. The work distribution unit 425 can track pending tasks received from the scheduler unit 420. In one embodiment, the work distribution unit 425 manages a pending task queue and an active task queue for each GPC 450. When a GPC 450 completes execution of a task, the task is evicted from the GPC's 450 active task queue and one of the other tasks from the pending task queue is selected and scheduled for execution on the GPC 450. If an active task on a GPC 450 has completed, for example, if the active task is waiting for a data dependency, then the active task can be evicted from the GPC 450 and returned to the pending task queue while another task from the pending task queue is selected and scheduled for execution on the GPC 450.

[0086] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations for execution on the PPU 400. In one embodiment, multiple compute applications are executed simultaneously by the PPU 400, and the PPU 400 provides isolation, quality of service (QoS), and independent address spaces for the multiple compute applications. An application can generate an instruction (e.g., an API call) that causes the driver kernel to generate one or more tasks for execution by the PPU 400. The driver kernel outputs the tasks to one or more streams that are being processed by the PPU 400. Each task can include one or more groups of related threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can be executed in parallel. Cooperative threads can refer to a plurality of threads that execute instructions of a task and can exchange data through shared memory. These tasks can be allocated to one or more processing units within a GPC 450, and instructions are scheduled for execution by at least one warp.

[0087] The work distribution unit 425 communicates with the one or more GPCs 450 via the XBar 470. The XBar 470 is an interconnect network coupling many units of the PPU 400 to other units of the PPU 400. For example, the XBar 470 can be configured to couple the work distribution unit 425 to a particular GPC 450. Although not explicitly shown, one or more other units of the PPU 400 can also be connected to the XBar 470 via the hub 430.

[0088] Tasks are managed by a scheduler unit 420 and dispatched to GPCs 450 by a work distribution unit 425. GPCs 450 are configured to process tasks and generate results. The results can be consumed by other tasks within GPCs 450, routed to different GPCs 450 via XBar 470, or stored in memory 404. Results can be written to memory 404 via a memory partition unit 480, which implements a memory interface for reading and writing data from / to memory 404. Results can be transmitted to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes a number U of memory partition units 480 equal to the number of separate and distinct package-on-package and off-chip combined memory systems coupled to PPU 400. Each GPC 450 can include a memory management unit to provide translations of virtual addresses into physical addresses, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more translation lookaside buffers (TLBs) for address translation and page walk operations. In one embodiment, the memory management unit translates virtual addresses into partition interface addresses and also logic base layer 130 in memory 404 performs the mapping. In other words, GPC 450 does not need to know where data is located, only that it is associated with one of U memory partition units 480.

[0089] In one embodiment, memory partition unit 480 includes a raster operations (ROP) unit, a level 2 (L2) cache, and a memory interface to memory 404. The memory interface can implement a 32-bit, 64-bit, 128-bit, 1024-bit data bus for high-speed data transfer. PPU 400 can connect to up to Y memory devices such as high bandwidth memory stacks or graphics double data rate, version 5, synchronous dynamic random access memory or other types of persistent storage. In one embodiment, the memory interface implements an HBM2 memory interface and Y is equal to half of U. In one embodiment, the HBM2 memory stacks are located on the same physical package as PPU 400, providing significant power and area savings compared to a conventional GDDR5 SDRAM system. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, with each HBM2 stack including two 129-bit channels per die for a total of 8 channels and a data bus width of 1024 bits.

[0090] In one embodiment, memory 404 supports single error correction dual error detection (SECDED) error correcting code (ECC) to protect data. ECC provides higher reliability for compute applications that are sensitive to data corruption. Reliability is particularly important in large scale cluster computing environments in which PPU 400 processes very large datasets and / or runs applications over extended periods.

[0091] In one embodiment, PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partition unit 480 supports a unified memory to provide a single unified virtual address space for CPU and PPU 400 memory, allowing data sharing between virtual memory systems. In one embodiment, the frequency at which PPU 400 accesses memory located on other processors is tracked such that memory pages that are frequently accessed on other PPU 400 are moved to the physical memory of the PPU 400 that is accessing those pages most frequently. In one embodiment, NVLink 410 supports an address translation service that allows PPU 400 to directly access CPU page tables and provide full access to CPU memory by PPU 400.

[0092] In one embodiment, a copy engine transfers data between multiple PPU 400 or between a PPU 400 and a CPU. The copy engine can generate a page fault for an address that is not mapped into a page table. Memory partition unit 480 can then service the page fault, map the address into the page table, and the copy engine can perform the transfer. In a conventional system, memory is pinned (e.g., unpageable) for multiple copy engine operations between multiple processors, greatly reducing the amount of available memory. With hardware page faults, an address can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0093] Data from memory 404 or other system memory can be fetched and stored by memory partition unit 480 in L2 cache 460, which is on-chip and shared among various GPCs 450. As shown, each memory partition unit 480 includes a portion of the L2 cache associated with the respective memory 404. Lower level caches can then be implemented within various units in the GPC 450. For example, each of the processing units within GPC 450 can implement a level one (LI) cache. The LI cache is private per processing unit and stores data and / or instructions cache-lined to the processing units. L2 cache 460 is coupled to the memory interface 470 and XBar 470, and data from the L2 cache can be fetched and stored in each of the LI caches for processing by the processing units.

[0094] In one embodiment, the processing units within each GPC 450 implement a SIMD (Single Instruction, Multiple Data) architecture wherein each thread in a group of threads (e.g., a warp) is configured to process a different element of a

[0095] A cooperative group is a programming model for organizing groups of threads that allows developers to express the granularity at which threads are communicating, allowing for a richer, more efficient expression of parallel decomposition. The Cooperative Launch API supports synchronization between thread blocks used to execute parallel algorithms. Conventional programming models provide a single simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often wish to define thread groups smaller than the thread block granularity in the form of collective group- scoped function interfaces, and synchronize within the defined groups in order to allow for greater performance, design flexibility, and software reuse.

[0096] Cooperative groups enable programmers to explicitly define thread groups at sub-block and multi-block granularity (as small as a single thread), and perform collective operations such as synchronization on threads in a cooperative group. This programming model supports clean composition across software boundaries, so that libraries and utility functions can safely synchronize within their local context without having to make assumptions about aggregation. Cooperative group primitives allow for new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire grid of thread blocks.

[0097] Each processing unit includes a large number (e.g., 128, etc.) of different processing cores (e.g., functional units) that can be fully pipelined, single-precision, double-precision, and / or mixed-precision, and include floating point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating point arithmetic logic units implement the IEEE 754-2008 standard for floating point arithmetic. In one embodiment, the cores include 64 single-precision (32-bit) floating point cores, 64 integer cores, 32 double-precision (64-bit) floating point cores, and 8 tensor cores.

[0098] The tensor cores are configured to perform matrix operations. In particular, the tensor cores are configured to perform deep learning matrix arithmetic, such as GEMM (matrix-matrix multiplication), for convolution operations during neural network training and inference. In one embodiment, each tensor core operates on 4x4 matrices and performs matrix multiplication and accumulation operations, D = A'B + C, where A, B, C, and D are 4x4 matrices.

[0099] In one embodiment, the matrix multiplication inputs A and B can be integer, fixed point, or floating point matrices, while the accumulation matrices C and D can be integer, fixed point, or floating point matrices of equal or higher bit-width. In one embodiment, the tensor cores operate on 1-bit, 4-bit, or 8-bit integer input data with 32-bit integer accumulation. An 8-bit integer matrix multiplication requires 1024 operations and results in a full precision product that is then accumulated with other intermediate products using 32-bit integer addition for an 8x8x16 matrix multiplication. In one embodiment, the tensor cores operate on 16-bit floating point input data with 32-bit floating point accumulation. A 16-bit floating point multiplication requires 64 operations and results in a full precision product that is then accumulated with other intermediate products using 32-bit floating point addition for a 4x4x4 matrix multiplication. In practice, the tensor cores are used to perform much larger two-dimensional or higher dimensional matrix operations composed of these smaller elements. APIs such as CUDA 9 C++ API expose specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use the tensor cores for CUDA-C++ programs. At the CUDA level, the interface at the warp level takes a 16x16 size matrix across all 32 threads of a warp.

[0100] Each processing unit can also include M special function units (SFUs) that perform special functions such as attribute evaluations, reciprocal square root, etc. In one embodiment, the SFUs can include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs can include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load texture maps (e.g., 2D texel arrays) from memory 404 and sample these texture maps to produce sampled texture values for use by a shader program executed by the processing unit. In one embodiment, the texture maps are stored in shared memory that can include or contain an LI cache. The texture unit uses mip maps (e.g., texture maps with varying levels of detail) to perform texture operations such as filtering operations. In one embodiment, each processing unit includes two texture units.

[0101] Each processing unit also includes N load store units (LSUs) that implement load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each of the cores to the register file and connects the LSUs to the register file, shared memory. In one embodiment, the interconnect network is a crossbar that can be configured to connect any of the cores to any of the registers in the register file and to connect the LSUs to registers in the register file and to memory locations in shared memory.

[0102] Shared memory is an on-chip memory array that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, shared memory includes 128 KB of storage capacity and is on a path from each of the processing units to the memory partition unit 480. Shared memory can be used for cache reads and writes. One or more of shared memory, LI cache, L2 cache, and memory 404 are backing stores.

[0103] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory accesses. The capacity can be used as a cache by programs that do not use shared memory. For example, if shared memory is configured to use half of the capacity, then the remaining capacity can be used for texture and load / store operations. Integration into shared memory enables shared memory to be used as a high throughput conduit for streaming data while providing high bandwidth and low latency access for frequently reused data.

[0104] When configured for general purpose parallel computing, a simpler configuration can be used compared to graphics computing. In particular, the fixed function graphics processing units are bypassed, creating a much simpler programming model. In this general purpose parallel computing configuration, the work distribution unit 425 dispatches and allocates thread blocks directly to the processing units within the GPCs 450. The threads execute the same program, using unique thread IDs, in the compute to ensure that each thread uses a processing unit on which to execute the program and perform the computation, shared memory to communicate between threads, and the LSU to read and write global memory through shared memory and the memory partitioning unit 480. When configured for general purpose parallel computing, the processing units can also write back commands that the scheduler unit 420 can use to initiate new work on the processing units.

[0105] Each of the PPU 400 can include one or more processing cores and / or components thereof, such as a tensor core (TC), a tensor processing unit (TPU), a pixel visual core (PVC), a ray tracing (RT) core, a visual processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI), or a peripheral component interconnect express (PCIe) element, and / or the like, configured to perform the functions thereof.

[0106] The PPU 400 can be included within a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smart-phone (e.g., a wireless, hand-held device), a personal digital assistant (PDA), a digital camera, a vehicle, a head mounted display, a hand-held electronic device, and / or the like. In one embodiment, the PPU 400 is implemented on a single semiconductor die. In another embodiment, the PPU 400 is included within a system-on-a-chip (SoC) along with, for example, one or more of the other devices, such as an additional PPU 400, a memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog front end (DAFE), and / or the like.

[0107] In one embodiment, PPU 400 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In yet another embodiment, PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in the chipset of the motherboard. In yet another embodiment, PPU 400 may be implemented in reconfigurable hardware. In yet another embodiment, a portion of PPU 400 may be implemented in reconfigurable hardware.

[0108] Exemplary computing system

[0109] As developers expose and leverage more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are being used across various industries. High-performance GPU-accelerated systems with tens to thousands of compute nodes are being deployed in data centers, research facilities, and supercomputers to solve increasingly complex problems. With the increasing number of processing devices within high-performance systems, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0110] Figure 5A According to one embodiment, the use Figure 4 A conceptual diagram of a processing system 500 implemented by a PPU 400. The exemplary system 500 can be configured to implement... Figure 1B Method 150 and / or shown in the figure Figure 3B The method 350 shown in the figure. The processing system 500 includes a CPU 530, a switch 510, multiple PPUs 400, and various memories 404.

[0111] The NVLink 410 provides a high-speed communication link between each of the PPUs 400. Although Figure 5B The diagram illustrates a specific number of NVLink 410 and interconnect 402 connections, but the number of connections to each PPU 400 and CPU 530 can vary. Switch 510 forms an interface between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols to interface between various connections and / or links.

[0112] In another embodiment (not shown), NVLinks 410 provide one or more high-speed communication links between each PPU 400 and CPU 530, and switch 510 forms an interface between interconnect 402 and each PPU 400. PPU 400, memory 404, and interconnect 402 can be located on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and switch 510 forms an interface between each PPU 400 using NVLinks 410 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLinks 410 provide one or more high-speed communication links between PPUs 400 and CPU 530 through switch 510. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 directly. One or more of the NVLink 410 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or on-die interconnects using the same protocol as NVLink 410.

[0113] In the context of this specification, a single semiconductor platform can refer to a singular integrated circuit die with all components integrated on the die, or a plurality of integrated circuit dies interconnected together with a substrate which can be a ceramic substrate or an interposer. In the context of this specification, a single semiconductor platform can also refer to a single package containing one or more integrated circuit dies, where a die can be a single chip or a stack of chips as will be appreciated by those of ordinary skill in the art. As will be appreciated by those of ordinary skill in the art, the various circuitry or

[0114] In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits / second, and each PPU 400 includes six NVLink 410 interfaces (as shown in FIG. 4B for each PPU 400 including five NVLink 410 interfaces). Figure 5A Each NVLink 410 provides a data transfer rate of 25 gigabytes / second in each direction, with six links providing 400 gigabytes / second. NVLinks 410 can be used exclusively for PPU-to-PPU communication, or for some combination of PPU-to-PPU and PPU-to-CPU when CPU 530 also includes one or more NVLink 410 interfaces. Figure 5A In

[0115] In one embodiment, the NVLink 410 allows direct load / store / atomic access from the CPU 530 to the memory 404 of each PPU 400. In one embodiment, the NVLink 410 supports coherency operations allowing data read from the memory 404 to be stored in the cache hierarchy of the CPU 530, reducing cache access latency for the CPU 530. In one embodiment, the NVLink 410 includes support for an address translation service (ATS) allowing the PPU(s) 400 to directly access page tables within the CPU 530. One or more of the NVLinks 410 can also be configured to operate in a low power mode.

[0116] Figure 5B FIGURE 13 illustrates an example system 565 in which various previous embodiments of the architecture and / or functionality can be implemented. The example system 565 can be configured to implement the method 150 illustrated in FIGURE Figure 1B and / or the method 350 illustrated in FIGURE Figure 3B .

[0117] As shown, a system 565 is provided that includes at least one central processing unit 530 coupled to a communication bus 575. The communication bus 575 can directly or indirectly couple one or more of the following: a main memory 540, a network interface 535, the CPU 530, a display device 545, an input device 560, a switch 510, and a parallel processing system 525. The communication bus 575 can be implemented using any suitable protocol, and can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The communication bus 575 can include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards board (VESA) bus, a peripheral component

[0118] Although the components of the system 565 are shown as discrete components, one of ordinary skill in the art will recognize that the components of the system 565 can be implemented as one or more sets of instructions executed by one or more processors (e.g., the CPU 530) of the system 565. Figure 5Bdifferent blocks are shown as being connected via a communication bus 575, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component such as display device 545 can be considered an I / O component, e.g., input device 560 if the display is a touch screen. As another example, CPU 530 and / or parallel processing system 525 can include memory (e.g., main memory 540 can represent a storage device in addition to parallel processing system 525, CPU 530, and / or other components). In other words, Figure 5B The computing device of FIG. 5 is merely illustrative. Distinctions are not made between Figure 5B different types of computing devices, such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "hand-held device", "game console", "electronic control unit (ECU)", "virtual reality system", and / or other device or system types, as all of these are contemplated within the scope of the computing device of FIG. 5.

[0119] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which can take various forms depending on the requirements of system 565. Computer-readable media can be any available media that can be accessed by system 565. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0120] Computer storage media can include volatile and non-volatile media, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules and / or other data types. For example, main memory 540 can store computer readable instructions such as an operating system (e.g., which represents programs and / or program elements). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by system 565. When used in the context herein, computer storage media does not include signals per se.

[0121] A computer storage medium can include a computer-readable instruction, a data structure, a program module, or other data type in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, a computer storage medium can include a wired medium such as a wired network or direct-wired connection, and a wireless medium such as sound, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0122] When executed, the computer program enables system 565 to perform various functions. CPU(s) 530 can be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Each of CPU(s) 530 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing multiple software threads concurrently. CPU(s) 530 can include any type of processors, and can include different types of processors depending on the type of system 565 being implemented (e.g., a fewer number of cores for mobile devices, and a greater number of cores for servers). For example, depending on the type of system 565, the processors can be Advanced RISC Machines (ARM) processors implemented using reduced instruction set computing (RISC) or x86 processors implemented using complex instruction set computing (CISC). System 565 can include one or more CPU(s) 530 in addition to one or more microprocessors or supplemental co-processors such as math co-processors.

[0123] In addition to or alternatively from CPU(s) 530, parallel processing module 525 can be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Parallel processing module 525 can be used by system 565 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, parallel processing module 525 can be used for general-purpose computing on GPUs (GPGPU). In embodiments, CPU(s) 530 and / or parallel processing module 525 can perform any combination of the described methods, processes, and / or portions thereof, discretely or jointly.

[0124] System 565 also includes input device 560, parallel processing system 525, and display device 545. Display device 545 can include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Display device 545 can receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output that data (e.g., images, video, sound, etc.).

[0125] Network interface 535 can enable system 565 to be logically coupled to other devices, including input device 560, display device 545, and / or other components, some of which can be embedded in (e.g., integrated with) system 565. Illustrative input devices 560 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. Input devices 560 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head tracking, and touch recognition (as described in more detail below) associated with a display of system 565. System 565 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition.

[0126] In addition, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) for communication purposes through network interface 535. System 565 can be included within a distributed network and / or cloud computing environment.

[0127] The network interface 535 can include one or more receivers, transmitters and / or transceivers that enable the system 565 to communicate with other computing devices via electronic communication networks including wired and / or wireless communication. The network interface 535 can include components and functionality allowing for communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0128] The system 565 can also include secondary storage (not shown). The secondary storage includes, for example, a hard disk drive and / or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory, etc. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner. The system 565 can also include a hard-wired or battery-backed-up

[0129] Each of the foregoing modules and / or devices can even be located on a single semiconductor platform. Alternatively, the various different modules can be located in separate devices or components and / or in various combinations thereof. Although the exemplary embodiments have been described above in terms of medical applications, those skilled in the art will recognize that the embodiments of the present disclosure also apply to other applications and that the systems and methods described herein have a wide range of applications.

[0130] Example Network Environment

[0131] A network environment suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on a processing system 500 and / or an example system 565, such as the example system 565 of FIG. 5, for example. Each device can include similar components, features, and / or functionality of the processing system 500 and / or the example system 565, for example. Figure 5A Figure 5B The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of the processing system 500 and / or the example system 565, such as the example system 565 of FIG. 5, for example. Each device can include similar components, features, and / or functionality of the processing system 500 and / or the example system 565, for example.

[0132] ​Components of a network environment can communicate with each other via a network, which can be wired, wireless, or a combination thereof. The network can include multiple networks or networks of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet, and / or the public switched telephone network (PSTN), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0133] A compatible network environment can include one or more peer-to-peer network environments, in which case servers can not be included in the network environment, and one or more client-server network environments, in which case one or more servers can be included in the network environment. In a peer-to-peer network environment, functionality described herein with respect to servers can be implemented on any number of client devices.

[0134] In at least one embodiment, a network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports a software layer and / or one or more applications of an application layer. The software or applications can include web-based service software or applications, respectively. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework type that, for example, can use a distributed file system for large-scale data processing (e.g., “big data”).

[0135] A cloud-based network environment can provide cloud computing and / or cloud storage that implement the computing and / or data storage functionality (or one or more portions thereof) described herein. Any of these different functionalities can be distributed across multiple locations from a central or core server (e.g., a central or core server of one or more data centers, which can be distributed across states, regions, countries, globally, and the like). If a connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0136] A client device can include Figure 5A at least some of the components, features, and functions of the example processing system 500 and / or Figure 5B The client device can be implemented as, include, or otherwise host an example system 565, at least some of the components, features, and functions of which are described herein. By way of example and not limitation, the client device can be implemented as, include, or otherwise host a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, a spaceship, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these delineated devices, or any other suitable device.

[0137] Machine learning

[0138] Deep neural networks (DNNs) developed on processors such as the PPU 400 have been used for a wide variety of use cases, from self-driving cars to faster drug development, from automatically captioning images in an online image database to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, learns continuously, gets smarter continuously, and delivers more accurate results more quickly over time. A child is initially taught by an adult to correctly identify and classify a variety of different shapes, and eventually is able to identify shapes without any guidance. Similarly, a deep learning or neural learning system needs to be trained in object recognition and classification so that it gets smarter and more efficient at identifying basic objects, occluded objects, and so on, while also imparting context to the objects.

[0139] At the simplest level, a neuron in the human brain looks at various inputs received, a level of importance is imparted to each of these inputs, and an output is delivered to other neurons to react. An artificial neuron or perceptron is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of an object that the perceptron is being trained to recognize and classify, and each of these features is imparted a certain weight based on the importance of that feature in defining the shape of the object.

[0140] Deep neural network (DNN) models include multiple layers of many connected nodes (e.g., perceptron, Boltzmann machine, radial basis function, convolutional layer, etc.) that can be trained with vast amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DNN model breaks down an input image of a car into different segments and looks for basic patterns such as lines and corners. The second layer assembles these lines to look for higher level patterns such as wheels, windshield, and mirrors. The next layer identifies the type of vehicle, and the final few layers generate a label for the input image that identifies the make and model of the particular car.

[0141] Once a DNN is trained, it can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks deposited into an ATM machine, identifying images of friends in a photograph, delivering movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in a self-driving car, or translating human languages in real time.

[0142] During training, data flows through the DNN in a forward propagation phase until a prediction is made that indicates a label corresponding to the input. If the neural network did not correctly label the input, the error between the correct label and the predicted label is analyzed and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels the input and other inputs in the training data set. Training complex neural networks requires a large amount of parallel computing performance, including floating point multiplication and addition supported by PPU 400. Inference, which is less computationally intensive than training, is a latency sensitive process in which a trained neural network is applied to new inputs that it has not seen before to classify images, detect emotions, identify recommendations, recognize and translate languages, and generally infer new information.

[0143] Neural networks rely heavily on matrix math operations, and for both efficiency and speed, complex multi-layer networks require vast amounts of floating point performance and bandwidth. With thousands of processing cores optimized for matrix math operations and providing tens to hundreds of TFLOPS of performance, PPU 400 is a computing platform capable of providing the performance needed for deep neural network based artificial intelligence and machine learning applications.

[0144] Further, images generated applying one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing objects and environments in the real world. Such images can include scenes of roadways, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real-world environments. Such images can be used to train, test, or certify DNNs employed in machines or robots for manipulating, handling, or modifying physical objects in the real world. Further, such images can be used to train, test, or certify DNNs employed in autonomous vehicles for navigating and moving the vehicles in the real world. Further, images generated applying one or more of the techniques disclosed herein can be used to convey information to users of such machines, robots, and vehicles.

[0145] Figure 5C Components of an example system 555 that can be used to train and utilize machine learning are shown in accordance with at least one embodiment. As will be discussed, various components can be provided by a single computing system or various combinations of computing devices and resources that can be under the control of a single entity or multiple entities. Further, various aspects can be triggered, initiated, or requested by different entities. In at least one embodiment, training of a neural network can be directed by a vendor associated with a vendor environment 506, while in at least one embodiment, training can be requested by a customer or other user that has access to the vendor environment through a client device 502 or other such resource. In at least one embodiment, training data (or data to be analyzed by a trained neural network) can be provided by a vendor, a user, or a third-party content provider 524. In at least one embodiment, a client device 502 can be, for example, a vehicle or object that is to be navigated on behalf of a user that can submit a request and / or receive instructions that facilitate navigation of the device.

[0146] In at least one embodiment, a request can be submitted through at least one network 504 for receipt by a vendor environment 506. In at least one embodiment, a client device can be any suitable electronic and / or computing device that enables a user to generate and send such a request, such as but not limited to a desktop computer, a notebook computer, a computer server, a smart phone, a tablet computer, a game console (portable or otherwise), a computer processor, computing logic, and a set-top box. One or more networks 504 can include any suitable network or networks for transmitting requests or other such data, for example, can include the Internet, an intranet, an Ethernet network, a cellular network, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), an ad hoc network of direct wireless connections between peers, and the like.

[0147] In at least one embodiment, a request can be received at interface layer 508, which in this example can forward data to training and inference manager 532. Training and inference manager 532 can be a system or service that includes hardware and software for managing services and requests corresponding to data or content. In at least one embodiment, training and inference manager 532 can receive a request to train a neural network, and can provide data for the request to training module 512. In at least one embodiment, training module 512 can select an appropriate model or neural network to use, if not specified by the request, and can train the model using relevant training data. In at least one embodiment, training data can be a batch of data stored in training data store 514, received from client device 502, or obtained from third party vendor 524. In at least one embodiment, training module 512 can be responsible for training data. A neural network can be any appropriate network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN). Once a neural network is trained and successfully evaluated, a trained neural network can be stored to, for example, model store 516, which can store different models or networks for users, applications, or services, etc. In at least one embodiment, there can be multiple models for a single application or entity, which can be utilized based on a number of different factors.

[0148] In at least one embodiment, at a subsequent point in time, a request for content (e.g., path determination) or data determined or influenced at least in part by a trained neural network can be received from client device 502 (or another such device). This request can include, for example, input data that is to be processed using a neural network to obtain one or more inference or other output values, classifications, or predictions, or input data can be received by interface layer 508 and directed to inference module 518, although different systems or services can also be used. In at least one embodiment, if not already stored locally to inference module 518, inference module 518 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 516. Inference module 518 can provide data as input to the trained network, which can then generate one or more inferences as output. For example, this can include a classification of an input data instance. In at least one embodiment, the inferences can then be transmitted to client device 502 for display to or other communication with a user. In at least one embodiment, a user’s contextual data can also be stored to user contextual data repository 522, which can include data about a user that can be used as network input to generate inferences or determine data returned to a user after an instance is obtained. In at least one embodiment, related data that can include at least some of input or inference data can also be stored to local database 534 for processing of future requests. In at least one embodiment, a user can use account information or other information to access resources or functionality of a vendor environment. In at least one embodiment, if permitted and available, user data can also be collected and used to further train models in order to provide more accurate inferences for future requests. In at least one embodiment, requests to machine learning application 526 executing on client device 502 can be received through a user interface and results displayed through the same interface. A client device can include resources such as a processor 528 and memory 562 for generating requests and processing results or responses, as well as at least one data storage element 552 for storing data for machine learning application 526. In one embodiment, memory 562 includes logic base layer 130 and memory system 255.

[0149] In at least one embodiment, processor 528 (or processor of training module 512 or inference module 518) will be a central processing unit (CPU). However, as described above, resources in such environments can utilize GPUs to process data for at least certain types of requests. GPUs, such as PPU 300, have thousands of cores designed to process large parallel workloads, and thus have become popular in deep learning for training neural networks and generating predictions. While using GPUs for offline building allows for training of larger, more complex models faster, offline generation of predictions means that request-time input features cannot be used, or must be prearranged for all features to generate predictions and store them in a lookup table to service real-time requests. If a deep learning framework supports CPU mode, and the model is small and simple enough to perform feedforward on a CPU with reasonable latency, a service on a CPU instance can host the model. In this case, training can be performed offline on a GPU, and inference performed in real-time on a CPU. If a CPU approach is not feasible, a service can run on a GPU instance. However, because GPUs have different performance and cost characteristics than CPUs, running a service that offloads runtime algorithms to a GPU can require designing it differently than a CPU-based service.

[0150] In at least one embodiment, video data can be provided from client device 502 for augmentation in vendor environment 506. In at least one embodiment, video data can be processed for augmentation on client device 502. In at least one embodiment, video data can be streamed from third party content vendor 524 and augmented by third party content vendor 524, vendor environment 506, or client device 502. In at least one embodiment, video data can be provided from client device 502 for use as training data in vendor environment 506.

[0151] In at least one embodiment, supervised and / or unsupervised training can be performed by client device 502 and / or vendor environment 506. In at least one embodiment, a set of training data 514 (e.g., classified or labeled data) is provided as input for use as training data.

[0152] In at least one embodiment, training data can include instances of at least one type of object for which a neural network is to be trained, and information identifying that object type. In at least one embodiment, training data can include a set of images each of which includes a representation of one type of object, where each image also includes or is associated with labeling, metadata, classification, or other information identifying the type of object represented in the corresponding image. Various other types of data can also be used as training data, which can include textual data, audio data, video data, and the like. In at least one embodiment, training data 514 is provided as training input to training module 512. In at least one embodiment, training module 512 can be a system or service including hardware and software, such as one or more computing devices executing a training application, for training a neural network (or other model or algorithm, etc.). In at least one embodiment, training module 512 receives instructions or requests indicating a type of model to be used for training, which in at least one embodiment can be any appropriate statistical model, network, or algorithm useful for such purposes, which can include artificial neural networks, deep learning algorithms, learning classifiers, Bayesian networks, and the like. In at least one embodiment, training module 512 can select an initial model or other untrained model from an appropriate repository, and train the model with training data 514, generating a trained model (e.g., a trained deep neural network) that can be used to classify or generate other such inferences on similar types of data. In at least one embodiment in which no training data is used, an initial model can still be selected for training on input data to each training module 512.

[0153] In at least one embodiment, a model can be trained in several different ways, which can depend in part on the type of model selected. In at least one embodiment, a machine learning algorithm can be provided with a training dataset, where the model is a model artifact created by a training process. In at least one embodiment, each instance of training data contains a correct answer (e.g., a classification) that can be referred to as a target or target attribute. In at least one embodiment, a learning algorithm finds patterns in the training data that map input data attributes to a target - the answer to be predicted - and the machine learning model is an output that captures these patterns. In at least one embodiment, a machine learning model can then be used to obtain predictions on new data for which a target is not specified.

[0154] In at least one embodiment, training and inference manager 532 can select from a set of machine learning models including binary classification, multiclass classification, generative, and regression models. In at least one embodiment, a type of model to be used can depend at least in part on a type of target to be predicted.

[0155] Graphics Processing Pipeline

[0156] In one embodiment, PPU 400 includes a graphics processing unit (GPU). PPU 400 is configured to receive commands that specify a shader program to be implemented for processing graphics data. The graphics data can be defined by a set of primitives such as points, lines, triangles, quads, triangle strips, etc. Typically, a primitive includes data specifying a number of vertices (e.g., in a model-space coordinate system) and attributes associated with each vertex of the primitive. PPU 400 can be configured to process the graphics primitives to generate a frame buffer (e.g., pixel data for each of the pixels of a display).

[0157] An application writes model data (e.g., attributes and a set of vertices) for a scene to a memory such as system memory or memory 404. The model data defines each of the objects that can be visible on a display. The application then makes an API call to a driver kernel that requests that the model data be rendered and displayed. The driver kernel reads the model data and writes commands to the one or more streams to perform operations that process the model data. The commands can reference different shader programs to be implemented on processing units within PPU 400, including one or more of a vertex shader, a hull shader, a domain shader, a geometry shader, and a pixel shader. For example, one or more of the processing units can be configured to execute a vertex shader program that processes a number of vertices defined by the model data. In one embodiment, the different processing units can be configured to execute different shader programs concurrently. For example, a first subset of processing units can be configured to execute a vertex shader program while a second subset of processing units can be configured to execute a pixel shader program. The first subset of processing units processes the vertex data to produce processed vertex data and writes the processed vertex data to L2 cache 460 and / or memory 404. After the processed vertex data is rasterized (e.g., transformed from three-dimensional data to two-dimensional data in screen space) to produce fragment data, the second subset of processing units executes a pixel shader to produce processed fragment data that is then blended with other processed fragment data and written to a frame buffer in memory 404. The vertex shader program and the pixel shader program can be executed concurrently, processing different data from the same scene in a pipelined fashion until all of the model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transferred to a display controller for display on a display device.

[0158] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device can be directly coupled to the system or processor that generates or renders the images. In other embodiments, the display device can be indirectly coupled to the system or processor, e.g., via a network. Examples of such networks include the Internet, mobile telecommunications networks, WIFI networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, the images generated by the system or processor can be streamed over the network to the display device. Such streaming allows, for example, video games or other applications that render images to be executed on a server, data center, or cloud-based computing environment, and the rendered images to be transmitted and displayed on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separate from the server or data center. Thus, the techniques disclosed herein can be applied to enhance images that are streamed, and to services that enhance streamed images, such as NVIDIA GeForce Now (GFN), Google Stadia, and the like.

[0159] Example streaming system

[0160] Figure 6 An example system diagram for a streaming system 605 in accordance with some embodiments of the present disclosure. Figure 6 The server 603 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The client 604 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The network 606 can be similar to the networks described herein. Figure 5A The server 603 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The client 604 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The network 606 can be similar to the networks described herein. Figure 5B The server 603 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The client 604 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The network 606 can be similar to the networks described herein. Figure 5A The server 603 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The client 604 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The network 606 can be similar to the networks described herein. Figure 5B The server 603 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The client 604 can include components, features, and / or functionality similar to those of the example processing system 500 and / or the example system 565 of FIG. 6, as described herein. The network 606 can be similar to the networks described herein.

[0161] In one embodiment, the streaming system 605 is a game streaming system, and the server 604 is a game server. In the system 605, for a game session, the client device 604 can receive only input data in response to input of the input device 626, send the input data to the server 603, receive encoded display data from the server 603, and display the display data on the display 624. In this way, more computationally intensive computations and processing are offloaded to the server 603 (e.g., rendering of graphical output of the game session, especially ray or path tracing, performed by the GPU 615 of the server 603). In other words, the game session is streamed from the server 603 to the client device 604, reducing the requirements of the client device 604 for graphics processing and rendering.

[0162] For example, with respect to instantiation of a game session, the client device 604 can be displaying a frame of the game session on the display 624 based on receiving display data from the server 603. The client device 604 can receive input from one of the input devices and, in response, generate input data. The client device 604 can send the input data to the server 603 via the communication interface 621 and over the network 606 (e.g., the Internet), and the server 603 can receive the input data via the communication interface 618. The CPU 608 can receive the input data, process the input data, and send data to the GPU 615 that causes the GPU 615 to generate a rendering of the game session. For example, the input data can represent movement of a user’s character in the game, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The rendering component 612 can render the game session (e.g., representing the results of the input data), and the rendering capture component 614 can capture the rendering of the game session as display data (e.g., as image data of a frame capturing the rendering of the game session). The rendering of the game session can include lighting and / or shadow effects computed using ray or path tracing of the server 603 (e.g., a GPU, which can further employ use of one or more specialized hardware accelerators or processing cores to perform ray or path tracing techniques). The encoder 616 can then encode the display data to generate encoded display data, and the encoded display data can be sent to the client device 604 via the communication interface 618 over the network 606. The client device 604 can receive the encoded display data via the communication interface 621, and the decoder 622 can decode the encoded display data to generate display data. The client device 604 can then display the display data via the display 624.

[0163] It should be noted that the techniques described herein can embody themselves in executable instructions stored in a computer readable medium for use by or in connection with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will recognize that a variety of computer readable media can be used to store data for some embodiments. As used herein, "computer readable medium" includes one or more of any suitable media for storing the executable instructions for a computer program so that an instruction execution machine, system, apparatus, or device can read (or fetch) the instructions from the computer readable medium and execute the instructions to implement the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer readable media includes: portable computer disks; random access memories (RAM); read only memories (ROM); erasable programmable read only memories (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CD), portable digital video discs (DVD), and so forth.

[0164] It should be understood that the arrangement of components shown in the figures is for illustrative purposes, and other arrangements are possible. For example, one or more of the elements described herein can be implemented in whole or in part as electronic hardware components. Other elements can be implemented in software, hardware, or a combination of software and hardware. Also, some or all of these other elements can be combined, some can be omitted entirely, and additional components can be added, while still implementing the functionality described herein. Accordingly, the subject matter described herein can be implemented in many different variations and all such variations are contemplated to be within the scope of the claims.

[0165] To facilitate understanding of the subject matter described herein, many aspects are described in the context of action sequences. Those skilled in the art will recognize that the various actions can be performed by specialized circuits or circuitry, by program instructions executed by one or more processors, or by a combination of both. The description of any sequence of actions herein does not necessarily imply that the particular order described for that sequence is the only order in which the sequence can be performed. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.

[0166] The use of the terms "one," "a," "the" and analogous expressions are used in the context of describing the subject matter herein (particularly in the context of the following claims) and are to be interpreted to cover both the singular and the plural unless otherwise indicated herein or clearly contradicted by the context. The use of the term "at least one" followed by a list of one or more items (for example, "at least one of A and B") is to be interpreted as meaning one item from the list A or B or any combination of two or more of the items in the list A and B, unless otherwise indicated herein or clearly contradicted by the context. Furthermore, the foregoing description is for the purpose of illustration only and not for the purpose of limitation, as the scope of the present application is defined by the claims as set hereinafter, together with their equivalents. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate the subject matter and does not pose a limitation on the scope of the subject matter unless otherwise claimed. The use of the "based on," along with other similar phrases (e.g., "based on the") indicating a condition for bringing about a result in both the claims and the written description is not intended to foreclose any other condition for bringing about that result. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

Claims

1. A system comprising: a processor coupled to an underlying layer within a package, the package enclosing the processor and the underlying layer; and the underlying layer including memory management logic coupled between the processor and a memory system, wherein the memory management logic processes memory access requests received from the processor, and wherein the memory system includes: at least one memory stack enclosed within the package and directly coupled to the underlying layer; and at least one memory device external to the package and coupled to the package, wherein a near memory pool includes a first location in the at least one memory stack and a second location in the at least one memory device, wherein the first location and the second location are interleaved within the near memory pool according to an address mapping performed by the memory management logic.

2. The system of claim 1, wherein the memory management logic translates each address associated with a memory access request according to an interface mapping to send the memory access request to one of the at least one memory stack or the at least one memory device.

3. The system of claim 1, wherein a far memory pool includes only locations in the at least one memory device.

4. The system of claim 1, wherein the memory management logic migrates first data stored in a third location in the at least one memory device that is not included in the near memory pool to the near memory pool in response to a trigger.

5. The system of claim 4, wherein portions of the first data are copied from the third location to the near memory pool while processing at least one additional memory access request.

6. The system of claim 4, wherein the memory management logic updates an address mapping for the first data to indicate that the first data is stored in the near memory pool.

7. The system of claim 4, wherein the memory management logic migrates data stored in the near memory pool to a fourth location in the at least one memory device that is not included in the near memory pool and updates an address mapping to indicate that the data is stored in the fourth location.

8. The system of claim 1, wherein a capacity of the at least one memory stack is less than a capacity of the at least one memory device.

9. The system of claim 1, wherein a data transfer rate of the at least one memory stack is greater than a data transfer rate of the at least one memory device.

10. The system of claim 1, wherein data stored in the at least one memory stack and the at least one memory device is sent over a connection between the processor and the underlying layer in response to a first memory access request.

11. The system of claim 1, wherein the at least one memory device is coupled to the base layer through a first interface, and the memory system further comprises at least one additional memory device that is external to the package and coupled to the base layer through a second interface that is separate from the first interface.

12. The system of claim 1, wherein at least a portion of the processor is included in a virtual machine.

13. The system of claim 1, wherein the processor is part of a server or data center, and data stored in the memory system is streamed to a user device.

14. The system of claim 1, wherein the processor generates data for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

15. The system of claim 1, wherein at least a portion of the processor is included in a cloud computing environment.

16. A base layer enclosed within a package, the base layer comprising: a first interface to directly couple a processor enclosed within the package to the base layer; memory management logic coupled between the interface and a memory system to process memory access requests received at the interface; at least one memory stack enclosed within the package and directly coupled to the base layer; and a second interface to send memory access requests to at least one memory device that is external to the package, wherein a near memory pool includes a first location in the at least one memory stack and a second location in the at least one memory device, wherein the first location and the second location are interleaved within the near memory pool according to an address mapping performed by the memory management logic.

17. The base layer of claim 16, wherein data is transferred between the memory management logic and both the at least one memory stack and the at least one memory device in response to a memory access request received from the processor.

18. A method comprising: receiving a memory access request from a processor coupled to a base layer within a package; and processing the memory access request by memory management logic fabricated within the base layer and coupled between the processor and a memory system by: sending a first portion of the memory access request to at least one memory stack enclosed within the package and directly coupled to the base layer; and sending a second portion of the memory access request to at least one memory device that is external to the package and coupled to the package through a first connection, wherein a near memory pool includes a first location in the at least one memory stack and a second location in the at least one memory device, wherein the first location and the second location are interleaved within the near memory pool according to an address mapping performed by the memory management logic. ​ ​ ​ 19. The method of claim 18, wherein the memory management logic translates each address associated with a memory access request according to an interface map to send the memory access request to one of the at least one memory stack or the at least one memory device.

Images