Memory Latency-Aware GPU Architecture

Abstract

One or more processing units [200, 205], such as graphics processing units (GPUs), execute applications. A resource manager [160] selectively allocates a first memory portion [130] or a second memory portion [135] to the processing units based on memory access characteristics. The first memory portion has a first latency that is lower than a second latency of the second memory portion. In some cases, the memory access characteristics indicate latency sensitivity. In some cases, hints included in corresponding program code are used to determine the memory access characteristics. The memory access characteristics can be determined by monitoring memory access requests, measuring a cache miss rate or a row buffer miss rate for the monitored memory access requests, and determining the memory access characteristics based on the cache miss rate or the row buffer miss rate.

Description

Background Art

[0001] Platforms for conventional processing systems include a central processing unit (CPU), a graphics processing unit (GPU), one or more memory modules (such as dynamic random access memory, i.e., DRAM, etc.), and a bus for supporting communication between these entities. In some cases, the platform is implemented as a monolithic processing architecture such as a system-on-chip (SoC). A distributed architecture provides an alternative to a monolithic processing architecture where physically or logically separated processing units operate cooperatively via high-performance interconnects. An example of a distributed architecture is a chiplet architecture, which gains the advantage of manufacturing some parts of the processing system in smaller nodes while allowing other parts to be manufactured in nodes with larger dimensions when other parts do not benefit from the reduced scale of the smaller nodes. Reducing the latency of the memory modules used by either the monolithic SOC architecture or the chiplet architecture improves the performance of applications running on the processing system. However, lower latency memory modules can incur higher costs and consume additional area. Therefore, there is a performance-cost-area trade-off to consider when selecting the type of memory to implement in a processing system.

[0002] The present disclosure will be better understood by reference to the accompanying drawings, and many of its features and advantages will become apparent to those of ordinary skill in the art. The use of the same reference numerals in different drawings indicates similar or identical items.

Brief Description of the Drawings

[0003] [Figure 1] FIG. [FIGURE NUMBER] is a block diagram of a processing system that selectively allocates low-latency memory to an application based on the memory access characteristics of the application, according to some embodiments. [Figure 2] Note: In the translation of the description of the figure (ID=17), the specific figure number in the original text is replaced with [FIGURE NUMBER] as it is not provided in the original. You may need to fill in the correct figure number according to the actual situation.This is a block diagram of a conventional graphics processing unit (GPU) implemented on a monolithic system-on-a-chip (SOC) according to several embodiments, and a monolithic SOC including a GPU that includes heterogeneous memory chips or memory stacks. [Figure 3] This is a block diagram of a traditional GPU chiplet architecture that includes one or more GPU chiplets. [Figure 4] This is a block diagram of a GPU chiplet architecture that implements heterogeneous memory chips or memory stacks according to several embodiments. [Figure 5] This is a block diagram of a processing system that performs online or offline scheduling for memory modules having different latencies based on profiles, according to several embodiments. [Figure 6] This is a flowchart illustrating a method for selectively allocating memory modules with different latencies based on the memory access characteristics of an application, according to several embodiments. [Modes for carrying out the invention]

[0004] A hierarchical heterogeneous memory architecture, or hybrid memory architecture, attempts to optimize memory latency by using different types of memory for different levels within a memory hierarchy. For example, the first level of the memory hierarchy can be implemented using relatively low-latency dynamic random-access memory (DRAM), while the second level can be implemented using relatively high-latency non-volatile RAM (NVRAM). The hierarchical memory architecture allocates the faster DRAM within the first level to store more frequently accessed memory data or hot pages. Data that is accessed less frequently is stored in the slower, larger NVRAM within the second level. Thus, the DRAM functions similarly to a cache for the NVRAM, and data is allocated to the DRAM based on the access characteristics associated with the memory location where the data is stored. A similar technique combines die-stacked DRAM with off-package DRAM within the same physical address space. One or more software layers are implemented to manage access to the two levels of DRAM based on the characteristics of memory access to the memory locations.

[0005] Low-latency memory offers different performance advantages to different types of applications running on a processing system. Applications that generate memory access requests with high locality do not derive significant benefits from low-latency memory because the regularity of memory access patterns increases the cache hit rate, which in turn reduces the need to access memory again. Furthermore, this improves the accuracy of algorithms that prefetch data from memory into the cache hierarchy. Thus, the majority of memory access requests are supplied from the cache hierarchy, and there are relatively few cache misses that require direct access to data from memory. In contrast, low-latency memory offers significant performance improvements to applications with irregular memory access patterns that make prefetching difficult. Low-latency memory also offers significant performance improvements to applications that do not generate memory access requests exhibiting high locality. These applications have a relatively large number of cache misses that require direct access to data from memory and benefit from the lower row cycle time of low-latency memory. Therefore, the performance-cost-area trade-off for implementing low-latency memory or standard memory in a processing system depends on the characteristics of the applications running on the processing system.

[0006] Figures 1 to 6 disclose an architecture that combines the performance benefits of low-latency memory with the cost and area benefits (e.g., relatively high compared to low-latency memory) of standard-latency memory in a processing system that implements heterogeneous memory, including memory with different latency characteristics. In some embodiments, a first memory portion provides a first latency lower than the second latency provided by a second memory portion within the heterogeneous memory. A resource manager selectively allocates memory in either the first or second memory portion to an application based on the application's memory access characteristics. Memory in the first memory portion is allocated to applications that generate memory access requests with low locality or irregular memory access patterns. Memory in the second memory portion is allocated to applications that generate memory access requests with relatively high locality. In some embodiments, the irregularity of memory access requests from an application is determined based on hints contained in the application's program code. In some embodiments, a resource manager monitors memory access requests from applications and measures statistics such as the cache miss rate for the monitored memory access requests. The resource manager then allocates (or reallocates) memory from the first or second memory portion based on statistical values. In some embodiments, the processing system is implemented as a monolithic processing unit including heterogeneous memory chips or stacks, including at least one low-latency memory chip or stack. In some embodiments, the processing system is implemented in a distributed architecture, such as a chiplet-based design. A first subset of chiplets includes low-latency memory, and a second subset of chiplets includes standard-latency memory.

[0007] Figure 1 is a block diagram of a processing system 100 that selectively allocates low-latency memory to an application based on the application's memory access characteristics, according to several embodiments. The processing system 100 includes, or has access to, memory 105 or other storage components implemented using non-temporary computer-readable media such as dynamic random access memory (DRAM), static random access memory (SRAM), or non-volatile RAM. The processing system 100 also includes a bus 110 for supporting communication between entities implemented in the processing system 100, such as memory 105. Some embodiments of the processing system 100 include other buses, bridges, switches, routers, etc., which are not shown in Figure 1 for clarity.

[0008] The processing system 100 includes at least one graphics processing unit (GPU) 115 that renders an image for presentation on the display 120. For example, the GPU 115 renders an object to generate pixel values ​​to be provided to the display 120, and the display 120 uses the pixel values ​​to display an image representing the rendered object. Some embodiments of the GPU 115 are used to implement DL operations, including CNNs, DNNs, and RNNs, and to perform other general-purpose computing tasks. In the illustrated embodiment, the GPU 115 includes an array 125 of compute units or processor cores that execute instructions for an application concurrently or in parallel. The GPU 115 also includes (or has access to) a memory module that includes a first memory portion 130 having a first latency and a second memory portion 135 having a second latency that is longer (e.g., has an increased duration) than the first latency. In the illustrated embodiment, the GPU 115 communicates with memory 105 via a bus 110. However, in some embodiments of the GPU 115, communication with memory 105 is made either directly or via other buses, bridges, switches, routers, etc. The GPU 115 executes instructions such as program code 138 stored in memory 105, and the GPU 115 stores information such as the results of the executed instructions in memory 105.

[0009] The processing system 100 also includes at least one central processing unit (CPU) 140 that implements one or more processor cores (not shown in Figure 1 for clarity) that execute instructions simultaneously or in parallel. The CPU 140 is connected to a bus 110 and therefore communicates with the GPU 115 and memory 105 via the bus 110. The CPU 140 executes instructions such as program code 145 stored in memory 105, and stores information such as the results of the executed instructions in memory 105. The CPU 140 can also initiate graphics processing by issuing a draw call to the GPU 115.

[0010] The input / output (I / O) engine 150 handles input or output operations associated with the display 120, as well as other elements of the processing system 100, such as a keyboard, mouse, printer, and external disk. The I / O engine 150 is coupled to a bus 110 so that it can communicate with memory 105, a GPU 115, or a CPU 140. In the illustrated embodiment, the I / O engine 150 reads information stored on an external storage component 155, which is implemented using a non-temporary computer-readable storage medium such as a compact disc (CD) or a digital video disc (DVD). The I / O engine 150 writes information such as the results of processing by the GPU 115 or CPU 140 to the external storage component 155.

[0011] Memory modules can be implemented using low-latency memory to help improve the performance of applications running on the GPU115. However, low-latency memory comes with higher area overhead and higher costs. The benefits of low-latency memory are not important for all applications due to the use of cache hierarchies and intelligent prefetching, and an optimized memory subsystem can limit memory access by keeping data within the chip used to implement the GPU115. On the other hand, some applications, such as those exhibiting irregular memory access and not row buffer locality, can enjoy significant benefits from reduced memory latency. A memory latency-aware GPU architecture combines the advantages of conventional and low-latency memory by scheduling latency-sensitive applications to run using low-latency memory and less latency-sensitive applications to run on conventional memory.

[0012] The GPU 115 implements a resource manager 160 that selectively allocates either a first memory portion 130 or a second memory portion 135 to an application based on its memory access characteristics. For example, the resource manager 160 can selectively allocate either the first memory portion 130 or the second memory portion 135 based on the memory access characteristics of an application running on an array 125 of processing units within the GPU 115. In another example, the resource manager 160 can selectively allocate either the first memory portion 130 or the second memory portion 135 based on the memory access characteristics of a kernel running on the GPU 115. In some cases, the memory access characteristics indicate the application's latency sensitivity to the kernel. Hints contained in the application or kernel's program code (e.g., program code 138 stored in memory 105) can be used to determine the memory access characteristics. Alternatively, the memory access characteristics can be determined by monitoring memory access requests from the application or kernel, measuring the cache miss rate or row buffer miss rate of the monitored memory access requests, and determining the memory access characteristics based on the cache miss rate or row buffer miss rate.

[0013] Some embodiments of the GPU115 are implemented as a monolithic system-on-a-chip (SOC) including heterogeneous memory chips or memory stacks. A first subset of the memory chips or stacks is used to implement a first memory portion 130, and a second subset of the memory chips or stacks is used to implement a second memory portion 135. Other embodiments of the GPU115 are implemented using a chiplet-based GPU design that includes a set of chiplets implementing corresponding coprocessors for implementing processing units and instances of the resource manager 160 for running applications. A first subset of the chiplets includes the first memory portion 130, and a second subset of the chiplets includes the second memory portion 135.

[0014] Figure 2 is a block diagram of a conventional GPU 200 implemented on a monolithic SOC 201 and a monolithic SOC 202 including a GPU 205 which includes heterogeneous memory chips or memory stacks, according to several embodiments. The SOC 202 including the GPU 205 is used to implement several embodiments of the GPU 115 shown in Figure 1. The conventional SOC 201 also includes memory modules such as high-bandwidth memory (HBM) modules 210, 215. In some embodiments, the HBM modules 210, 215 are implemented as a three-dimensional (3D) circuit stacking multiple memory dies, such as dynamic random-access memory (DRAM) dies. The HBM module 210 may also include a base die having a memory controller interconnected with the stacked memory dies using through-silicon vias and microbumps. The memory within the HBM modules 210, 215 exhibits a first latency, as defined herein, which is the time required to perform a memory access operation, such as reading a value in a memory cell or writing a value to a memory cell. The conventional SOC201 further includes a coprocessor 220 that complements the GPU205's capabilities by offloading processor-intensive tasks such as floating-point arithmetic, signal processing, string processing, cryptography, and I / O interfaces.

[0015] The SOC202, which implements the GPU205, includes heterogeneous memory modules with different latencies. In the illustrated embodiment, the SOC202 includes an HBM module 225 with standard latency and a low-latency HBM (LL-HBM) module 230 with lower latency than standard latency. The SOC202 further includes a coprocessor 235 that complements the functionality of the GPU205. In the illustrated embodiment, the coprocessor 235 runs an instance of a heterogeneous resource manager (HRM) 240, although some embodiments of the SOC202 implement the HRM240 as separate modules such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). The HRM240 selectively allocates portions of the HBM225 and LL-HBM230 to applications running on the GPU205 based on memory access characteristics, such as the memory access characteristics of the kernel or application running on the GPU205. In some embodiments, the HRM240 determines the memory access characteristics of the kernel or application based on hints contained in the corresponding program code. In some embodiments, the HRM240 determines the memory access characteristics by monitoring memory access requests to the HBM225 or LL-HBM230, as described herein.

[0016] Figure 3 is a block diagram of a conventional GPU chiplet architecture 300, which includes one or more GPU chiplets such as GPU chiplets 301, 302, 303, 304, etc., collectively referred to herein as “GPU chiplets 301-304”. As used herein, the term “chiplet” refers to any device having the following characteristics: 1) A chiplet includes an active silicon die containing at least a portion of the computational logic used to solve a complete problem (i.e., the computational workload is distributed across multiple of these active silicon dies); 2) Chiplets are formed on the same substrate and packaged together as a monolithic unit; 3) The programming model maintains the concept that the combination of these separate computational dies (i.e., GPU chiplets 301-304) is a single monolithic unit. Thus, GPU chiplets 301-304 are not exposed as separate devices to applications that use GPU chiplets 301-304 to handle computational workloads. Four GPU chiplets 301-304 are shown in Figure 3, but the number of GPU chiplets in the GPU chiplet architecture 300 is a design choice and may differ in other embodiments.

[0017] The GPU chiplets 301-304 include corresponding memory modules such as HBM311, 312, 313, 314, 315, 316, 317, and 318, collectively referred to herein as "HBM311-318". In the illustrated embodiment, HBM311-318 operate with a first memory access latency. The GPU chiplets 301-304 also include corresponding GPUs 321, 322, 323, and 324 (collectively referred to herein as "GPU321-324") and coprocessors 331, 332, 333, and 334 (collectively referred to herein as "coprocessors 331-334"). The I / O die 340 coordinates the provision of information to the GPU chiplets 301-304, the transmission of information from the GPU chiplets 301-304 to other entities such as the CPU 140 shown in Figure 1, and the exchange of information between the GPU chiplets 301-304. For clarity, the numerous interconnections between the I / O die 340, the GPU chiplets 301-304, and the external entities are not shown in Figure 3.

[0018] Figure 4 is a block diagram of a GPU chiplet architecture 400 implementing heterogeneous memory chips or memory stacks in several embodiments. The GPU chiplet architecture 400 includes one or more GPU chiplets, such as GPU chiplets 401, 402, 403, 404, etc., collectively referred to herein as "GPU chiplets 401-404". Although four GPU chiplets 401-404 are shown in Figure 4, the number of GPU chiplets in the GPU chiplet architecture 400 is a design choice and will differ in other embodiments.

[0019] GPU chiplets 401, 402, and 404 include corresponding memory modules such as HBM411, 412, 413, 414, 415, and 416, collectively referred to herein as "HBM411-418". In the illustrated embodiments, HBM411-416 operate with a first memory access latency. GPU chiplet 403 includes low-latency memory modules such as LL-HBM417 and 418, which operate with a second memory access latency shorter than the first memory access latency. GPU chiplets 401-404 also include corresponding GPUs 421, 422, 423, and 424 (collectively referred to herein as "GPU421-424") and coprocessors 431, 432, 433, and 434 (collectively referred to herein as "coprocessors 431-434"). The I / O die 440 coordinates the provision of information to the GPU chiplets 401-404, the transmission of information from the GPU chiplets 401-404 to other entities such as the CPU 140 shown in Figure 1, and the exchange of information between the GPU chiplets 401-404. For clarity, the numerous interconnections between the I / O die 440, the GPU chiplets 401-404, and the external entities are not shown in Figure 4.

[0020] Coprocessor 434 runs an instance of HRM445, while some embodiments of the GPU chiplet 404 implement HRM445 as a separate module such as an ASIC or FPGA. In some embodiments, other instances of HRM are run by coprocessors 431-433, and the other instances coordinate their operation with HRM445. HRM445 (in cooperation with other instances of HRM, if present) selectively schedules kernels or applications to run on chiplets 401-404 based on the memory access characteristics of the application and the latency of the memory implemented on chiplets 401-404. In the illustrated embodiment, HRM445 (in cooperation with other instances of HRM, if present) schedules kernels or applications to run on chiplets 404, including LL-HBM417,418, depending on whether it has determined that the application exhibits irregular memory access requests, or a combination thereof, with a relatively high cache miss rate, a relatively high row buffer miss rate, or requires relatively low latency. Otherwise, HRM445 (in cooperation with other instances of HRM, if any) schedules a kernel or application to run on one of the chiplets 401-403. In some embodiments, HRM445 (in cooperation with other instances of HRM, if any) determines the memory access characteristics of an application based on hints contained in the application's program code. In some embodiments, HRM445 (in cooperation with other instances of HRM, if any) determines the memory access characteristics of an application by monitoring memory access requests to HBM411-416 or LL-HBM417,418, as described herein.

[0021] As described herein, in some embodiments, an application is selectively scheduled for execution on a GPU associated with a memory having a relatively high or low latency based on information included in the application's program code. For example, a programmer can annotate the code in a way that provides hints regarding the memory access characteristics of a kernel. This approach can complement the HRM that obtains this information and allocates / schedules accordingly. Often, the programmer has sufficient knowledge of the underlying complexity of the algorithm to give hints to the compiler and runtime as to where to appropriately schedule the algorithm. In some embodiments, language extensions (e.g., pragmas or modifiers) are provided in the hints so that the compiler can annotate the kernel.

[0022] The following are examples of two HIP kernels. The first kernel contains information indicating regular memory access behavior, and the second kernel contains information indicating irregular memory access. _global_ void regular_kernel(int *a,int *b){...} _low_latency_memory_ _global_ void irregular_kernel(int *x,int *y){...} Similarly, data objects accessed by these irregular kernels can be marked via language extensions or the memory allocator can be changed to target a particular GPU's memory stack. _low_latency_memory_foo x; bar*y=low_latency_malloc(sizeof(bar)); Including hints in the program code requires the programmer to make code changes and, in some cases, requires the programmer to have a deep understanding of the algorithm being executed.

[0023] In some embodiments, instead of performing memory allocation in software at runtime, kernels or applications are scheduled using dedicated hardware threads. The hardware monitors the profile statistics of kernels or applications running on the GPU and then schedules the kernels or applications to appropriate memory based on the profile statistics. This approach results in a simplified software stack and potentially better performance by sacrificing additional hardware to reduce software overhead.

[0024] Off-line profile-guided scheduling uses an off-line profile of kernel memory access behavior to schedule irregular kernels and allocate data to a GPU with lower latency memory. The profiler reads the statistics associated with the running kernel. The statistics represent operations such as how frequently the kernel is causing cache misses and row buffer misses in memory. The profiling results are fed into a runtime scheduling algorithm, which allocates kernels with specific characteristics to a GPU module with lower latency memory. The memory allocation of data associated with these kernels is guided through this profile and can be implemented with HRM. Off-line profile-guided scheduling should require little or no programmer intervention and should not have a very high performance overhead during execution. However, this approach requires a profiling stage and decisions are made based on the profiled input. Thus, the actual input during execution can change the behavior of the algorithm and modify the memory access behavior.

[0025] Online dynamic scheduling allows the HRM to read performance counters and make dynamic scheduling decisions while an application is running. When a kernel is running, the HRM collects statistics and tracks kernels that have previously shown memory characteristics such as high miss rates in low buffers or caches, which can indicate that the kernel is likely to benefit from scheduling on low-latency memory. Memory allocation is then adjusted based on these statistics. Kernel and memory objects can be dynamically migrated to the GPU(s) with low-latency memory. While online dynamic scheduling adapts to the behavior of running applications, it incurs higher overhead in the HRM scheduler because it requires the collection of statistics to make decisions. Furthermore, significant overhead can exist due to data movement when kernels and data are migrated to different chiplets or memory locations.

[0026] Figure 5 is a block diagram of a processing system 500 that performs online or offline scheduling for memory modules with different latencies based on profile 505, according to several embodiments. The processing system 500 is used to implement several embodiments of the SOC 202 shown in Figure 2 and the GPU chiplet architecture 400 shown in Figure 4. In the illustrated embodiments, the processing system 500 includes a GPU 510, an HBM 515, and an LL-HBM 520 having lower latency than the HBM 515. The GPU 510 runs applications that are selectively scheduled by the HBM 525 to use either the HBM 515 or the LL-HBM 520, based on the memory access characteristics of the application.

[0027] Information indicating memory access characteristics is stored in profile 505. As described herein, profile 505 can be an offline profile populated before application execution (e.g., based on hints in program code) or an online profile populated dynamically concurrently with application execution. In some embodiments, the information in profile 505 represents the required memory access latency or the degree of irregularity of memory access requests. For example, HRM 525 can monitor the miss rate or other buffer locality statistics for row buffers associated with HBM 515 or LL-HBM 520 to determine whether memory access requests are relatively regular or irregular. Information representing the miss rate is stored in profile 505, and the stored information is used to selectively schedule the application to HBM 515 (when the row buffer miss rate is relatively low and memory access requests are relatively regular) or LL-HBM 520 (when the row buffer miss rate is relatively high and memory access requests are relatively irregular).

[0028] Figure 6 is a flowchart of method 600, which selectively allocates memory modules with different latencies based on the memory access characteristics of an application, according to several embodiments. Method 600 is implemented in several embodiments of the SOC202 shown in Figure 2 and the GPU chiplet architecture 400 shown in Figure 4.

[0029] In block 605, the memory access characteristics of the application are determined. In some embodiments, as described herein, the HRM determines the memory access characteristics based on one or more of the following: hints contained in the program code, offline profiles, or dynamically generated online profiles.

[0030] In decision block 610, HRM determines whether the memory access characteristics indicate that the application is latency sensitive. For example, HRM may determine whether it is likely to receive performance improvements from scheduling memory access requests to low-latency memory. If so, method 600 proceeds to block 615. Otherwise, method 600 proceeds to block 620.

[0031] In block 615, low-latency memory is allocated to the application. In block 620, memory other than low-latency memory is allocated to the application.

[0032] As disclosed herein, in some embodiments, the apparatus includes at least one processing unit and a resource manager configured to selectively allocate a first memory portion or a second memory portion to at least one processing unit based on memory access characteristics, wherein the first memory portion has a first latency lower than the second latency of the second memory portion. In one embodiment, the memory access characteristics represent the latency sensitivity of the application. In another embodiment, the resource manager allocates the first memory portion in response to a memory access request having low locality or an irregular memory access pattern. In yet another embodiment, the resource manager allocates the second memory portion in response to a memory access request having relatively high locality or an irregular memory access pattern. In yet another embodiment, the irregularity of memory access request n is determined based on hints contained in the corresponding program code.

[0033] In one embodiment, the resource manager is configured to monitor memory access requests and measure statistics for memory access requests. In another embodiment, the statistics represent the cache miss rate or row buffer miss rate for the monitored memory access requests. In yet another embodiment, the resource manager is configured to allocate or reallocate a first memory portion or a second memory portion based on the statistics. In yet another embodiment, the device includes at least one heterogeneous memory chip or heterogeneous memory stack having a first memory portion and a second memory portion. In yet another embodiment, the device includes a plurality of chiplets, each comprising a plurality of processing units and a plurality of coprocessors configured to implement instances of the resource manager, wherein a first subset of the chiplets comprises a first memory portion and a second subset of the chiplets comprises a second memory portion.

[0034] In some embodiments, the method includes the steps of: selectively allocating a first memory portion or a second memory portion to at least one processing unit based on memory access characteristics, wherein the first memory portion has a first latency lower than a second latency of the second memory portion; and executing at least one application or kernel on at least one processing unit using the allocated first or second memory portion. In one embodiment, selectively allocating a first memory portion or a second memory portion includes allocating the first memory portion in response to a memory access request having low locality or an irregular memory access pattern. In another embodiment, selectively allocating a first memory portion or a second memory portion includes allocating the second memory portion in response to a memory access request having relatively high locality or an irregular memory access pattern. In yet another embodiment, the method includes the step of determining the irregularity of a memory access request from an application based on hints contained in the corresponding program code.

[0035] In one embodiment, the method includes the steps of monitoring memory access requests and measuring statistics of the memory access requests. In another embodiment, measuring statistics includes measuring the cache miss rate or row buffer miss rate of the monitored memory access requests. In yet another embodiment, the method includes allocating or reallocating a first memory portion or a second memory portion based on the statistics.

[0036] In some embodiments, the method includes the steps of: selectively allocating a first memory portion or a second memory portion to at least one processing unit based on latency sensitivity, wherein the first memory portion has a first latency lower than a second latency of the second memory portion; and executing at least one application or kernel on at least one processing unit using the allocated first or second memory portion. In another embodiment, the method includes the step of determining latency sensitivity based on hints contained in the corresponding program code. In yet another embodiment, the method includes the steps of: monitoring memory access requests; measuring the cache miss rate or row buffer miss rate of the monitored memory access requests; and determining latency sensitivity based on the cache miss rate or row buffer miss rate.

[0037] In some embodiments, the apparatus and techniques described above are implemented in a system including one or more integrated circuit (IC) devices (also called integrated circuit packages or microchips), such as the GPU architecture described with reference to Figures 1 to 6. Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used to design and manufacture these IC devices. These design tools are typically represented as one or more software programs. One or more software programs include computer-executable code for operating a computer system to operate with code representing the circuit of one or more IC devices in order to perform at least part of the process of designing or adapting a manufacturing system for manufacturing the circuit. This code may include instructions, data, or combinations of instructions and data. Software instructions representing the design or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of designing or manufacturing an IC device is stored in and accessed from the same or different computer-readable storage medium.

[0038] Computer-readable storage media include any non-temporary storage media or combination of non-temporary storage media that are accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray® discs), magnetic media (e.g., floppy disks, magnetic tapes, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS) based storage media. Computer-readable storage media (e.g., system RAM or ROM) may be built into the computing system, computer-readable storage media (e.g., magnetic hard drives) may be permanently mounted to the computing system, computer-readable storage media (e.g., optical disks or Universal Serial Bus (USB) based flash memory) may be detachably mounted to the computing system, and computer-readable storage media (e.g., network-accessible storage (NAS)) may be connected to the computer system via a wired or wireless network.

[0039] In some embodiments, certain aspects of the technology described above are implemented by one or more processors of a processing system that executes the software. The software includes one or more sets of executable instructions, which are stored in a non-temporary computer-readable storage medium or otherwise clearly embodied. The software may also include instructions and specific data, which, when executed by one or more processors, operate the one or more processors to execute one or more aspects of the technology described above. Non-temporary computer-readable storage mediums may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, caches, random-access memory (RAM), or other non-volatile memory devices (one or more). Executable instructions stored in a non-temporary computer-readable storage medium can be implemented as source code, assembly language code, object code, or other instruction forms that can be interpreted or otherwise executed by one or more processors.

[0040] In addition to the foregoing, it should be noted that not all activities or elements described in the summary are required, and certain activities or parts of devices may not be required, and one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed does not necessarily indicate the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and variations can be made without departing from the scope of the invention as described in the claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive, and all of these variations are intended to fall within the scope of the invention.

[0041] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and features that may give rise to or manifest any benefits, advantages, or solutions are not to be construed as essential, necessary, or indispensable features to any or all of the claims. Furthermore, the disclosed invention can be modified and implemented in different but similar ways, in a manner that is obvious to those skilled in the art who are interested in the teachings of this specification; therefore, the specific embodiments described above are merely illustrative. There are no limitations to the details of the configuration or design shown herein beyond those described in the appended claims. Accordingly, the specific embodiments described above may be modified or altered, and it is clear that all such modifications are within the scope of the disclosed invention. Accordingly, the protection sought herein is described in the appended claims.

Claims

1. It is a device, A plurality of chiplets, wherein a first subset of the plurality of chiplets comprises a first memory portion, and a second subset of the plurality of chiplets comprises a second memory portion, At least one processing unit that runs an application or kernel, The system includes a resource manager configured to selectively allocate the first memory portion or the second memory portion to at least one processing unit based on memory access characteristics indicating the latency sensitivity of the application or the kernel, The first memory portion has a first latency that is lower than the second latency of the second memory portion. Device.

2. The resource manager allocates the first memory portion to the at least one processing unit based on whether the application has a high cache miss rate, a high row buffer miss rate, or exhibits irregular memory access requests. The apparatus according to claim 1.

3. The resource manager allocates the first memory portion in response to memory access requests having low locality or irregular memory access patterns. The apparatus according to claim 1.

4. The resource manager allocates the second memory portion in response to the memory access request having a relatively high locality or regular memory access pattern. The apparatus according to claim 3.

5. The irregularity of the memory access request is determined based on hints contained in the corresponding program code. The apparatus according to claim 3.

6. The resource manager is configured to monitor memory access requests and measure statistics of those memory access requests. The apparatus according to any one of claims 1 to 5.

7. The aforementioned statistical values ​​represent the cache miss rate or row buffer miss rate of the monitored memory access requests. The apparatus according to claim 6.

8. The resource manager is configured to allocate or reallocate the first memory portion or the second memory portion based on the statistical values. The apparatus according to claim 6.

9. The plurality of chiplets comprises a plurality of processing units and a plurality of coprocessors configured to implement instances of the resource manager. The apparatus according to claim 1.

10. It is a method, A step of selectively allocating a first memory portion or a second memory portion to at least one processing unit based on memory access characteristics indicating the latency sensitivity of an application or kernel running on at least one processing unit, wherein the first memory portion has a first latency lower than the second latency of the second memory portion, a first subset of a plurality of chiplets comprises the first memory portion, and a second subset of the plurality of chiplets comprises the second memory portion. The step of executing at least one of the application or the kernel using the allocated first memory portion or the second memory portion in the at least one processing unit, method.

11. The step of selectively allocating the first memory portion or the second memory portion includes the step of allocating the first memory portion in response to a memory access request having low locality or an irregular memory access pattern. The method of claim 10.

12. The step of selectively allocating the first memory portion or the second memory portion includes the step of allocating the second memory portion in response to a memory access request having relatively high locality or a regular memory access pattern. The method according to claim 11.

13. The further step includes determining the irregularity of memory access requests from the application based on hints contained in the corresponding program code. The method according to claim 11 or 12.

14. Steps to monitor memory access requests, The further step includes measuring statistical values ​​for the memory access request, The method according to any one of claims 10 to 13.

Images