Adaptive Caching of Memory Request Streams

Abstract

In one embodiment, the present invention relates to a method for allocating cache resources based on a stream ID, the method comprising: caching a memory request on each of one or more integrated client devices;

Description

[Technical Field]

[0001] The present disclosure relates to systems having integrated circuit devices. [Background technology]

[0002] A cache is an auxiliary device that manages data traffic to memory. A cache interacts with one or more hardware devices in a system to store data retrieved from memory, data to be written to memory, or both. The hardware devices may be various components of an integrated circuit and may be implemented in a system-on-chip (SOC). Devices that service read and write requests through a cache or directly to memory are called client devices.

[0003] Caches are frequently utilized to reduce power consumption by limiting the total number of requests to main memory. Further power savings can be achieved by placing main memory and the data paths to main memory in a low-power state. Due to the inverse correlation between cache usage and power consumption, maximizing cache usage reduces overall power consumption. The power capacity of battery-powered devices, such as mobile computing devices, can be consumed more efficiently by increasing cache usage in integrated client devices. Furthermore, accessing a cache is generally faster than accessing main memory, thereby improving the performance of integrated client devices.

[0004] Caches are typically organized into partitions to increase cache usage. A partition represents a portion of the cache allocated for a specific purpose or to a specific entity, such as a specific client device. However, effective cache partitioning and stream allocation can be challenging due to the limited size of partitions relative to the working data set of a system, such as a mobile computing device, and the way memory needs shift over time. Cache thrashing can occur when memory requests from client devices compete for the same resources within their respective partitions, thus reducing cache usage. In these cases, system operation may not progress, thereby reducing system performance and increasing power consumption. Therefore, maximizing cache usage depends on optimizing stream allocation to cache partitions. Summary of the Invention

[0005] This specification describes techniques for implementing caching policies in caches driven by correlated data streams, referred to herein as "computing tasks." As used herein, a computing task can be associated with multiple interrelated memory requests in software. For example, a computing task can include all requests for data or all requests for instructions to a client device (or software driver). Depending on the particular workload of a client device, the device may execute multiple computing tasks, serially or in parallel, each involving many related memory requests.

[0006] The cache can identify computing tasks by examining the stream IDs that various memory requests have in common. The cache can then assign different partitions of the cache memory to different tasks by referencing their respective stream IDs. Thus, for example, instruction requests can be allocated to a different partition of the cache than data requests. Furthermore, the cache can adaptively assign computing tasks based on each partition's corresponding hit metric. This feature allows the cache to self-tune for an optimal assignment of computing tasks that maximizes cache performance.

[0007] Particular embodiments of the subject matter described herein can be implemented to realize one or more of the following advantages: The cache can improve cache performance and utilization by using the stream ID to determine associated computing tasks. The cache can therefore reduce contention for cache resources for different computing tasks, increasing the cache hit rate. Increasing the cache hit rate reduces power consumption and extends battery life in mobile devices that rely on battery power. The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of an exemplary system. [Figure 2] FIG. 2 is a diagram of an exemplary subsystem. [Figure 3] 1 is a flowchart of an exemplary process for allocating partitions in a cache. [Figure 4]1 is a flowchart of an exemplary process for serving memory requests using a partition of a cache dedicated to a computing task. [Figure 5] 1 is a flowchart of an example process for adaptively allocating computing tasks to partitions of a cache. DETAILED DESCRIPTION OF THE INVENTION

[0009] Like reference symbols and designations in the various drawings refer to like elements. FIG. 1 is a diagram of an exemplary system 100. System 100 includes several client devices 110a, 110b, . . . 110n that service memory requests to locations in memory device 140. The aforementioned components may be integrated onto a single system-on-chip (SOC) 102. Memory controller 130 may process data requests to and from memory device 140 of system 100. Cache 120 caches data requests for multiple client devices on SOC 102; therefore, cache 120 may be referred to as a system-level cache (SLC). However, the techniques described below may be utilized for various types of devices that perform caching of memory requests. For example, a cache that caches memory requests for a single client device or software driver, or a cache that caches memory requests for client devices that are not integrated on the same SOC 102 as cache 120.

[0010] SOC 102 is an example of a device that can be installed on or integrated into any suitable computing device, which may be referred to as a host device. Because the techniques described herein are particularly well-suited for reducing power consumption and improving performance of the host device, SOC 102 can be particularly beneficial when installed in a mobile host device that relies on battery power, such as a smartphone, a smartwatch or another wearable computing device, a tablet computer, or a laptop computer, to name a few.

[0011] SLC 120 is an example of a cache that can be partitioned. Partitions 112a-n are portions, such as ways or sets, within the cache that are allocated to memory requests having one or more attributes.

[0012] Multiple client devices 110a-n are integrated on SOC 102. Each of client devices 110a-n may be a suitable module, device, or functional component configured to communicate memory requests to cache 120 and memory controller 130 via SOC fabric 150. For example, client devices 110a-n, or SOC 102 itself, may be a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an ambient computing module, an image processor, a sensor processing module, an application-specific integrated circuit (ASIC), or other low-level component of SOC 102 itself that can issue memory requests to memory controller 130 via SOC fabric 150.

[0013] Client devices 110a-n submit memory requests through SOC fabric 150 in the course of implementing workloads, with each workload being executed by one or more computing tasks 112a-n. Each computing task may have one or more threads that execute on the client device. Thus, computing tasks and / or threads may be associated with particular memory requests submitted by client devices 110a-n.

[0014] The exemplary diagram of FIG. 1 illustrates the workload of each client device with computing tasks 112a-n having different stream IDs. In this case, the stream ID uniquely identifies a particular computing task. The stream ID can be any identifier that distinguishes computing tasks, such as a universally unique identifier (UUID). Although not shown in FIG. 1, threads of each computing task can also have stream IDs. Thus, a memory request can include one or more stream IDs that identify the computing task (or thread) to which the memory request belongs.

[0015] The SOC 102, or client devices 110a-n on the SOC 102, can pre-assign stream IDs to specific tasks 112a-n so that the stream IDs are included in memory requests from those tasks. For example, in the case of a GPU client device running a workload with many threads, each thread can have a separate pre-assigned stream ID. In some implementations, the stream IDs assigned to memory requests from the SOC 102 are based on the type of task being performed. As an example, a GPU thread responsible for texture mapping can be pre-assigned to have a different stream ID than a thread responsible for rendering polygons. As another example, all threads in a TPU running on the same layer of a neural network may share data with each other. These threads can be associated with a common computing task and pre-assigned their own stream IDs. Other threads running on different layers can be associated with their own computing tasks and their own stream IDs. In this context, pre-assigned stream IDs mean that a stream ID is assigned to a computing task before the SLC 120 begins processing the computing task's memory request.

[0016] More generally, any related set of memory requests can have a corresponding stream ID. The stream IDs for memory requests can be programmed, preloaded into client devices 110a-n, specified by cache 120 or SOC fabric 150, or dynamically created while responding to memory requests. In some cases, a set of memory requests can include memory requests from multiple client devices associated with respective stream IDs.

[0017] SOC fabric 150 is the communications subsystem of SOC 102. SOC fabric 150 includes communication paths that allow client devices 110a-n to communicate with each other and to request data reads and writes using cache 120 and memory controller 130. SOC fabric 150 may include any suitable combination of communications hardware, such as, for example, a bus or dedicated interconnect circuitry.

[0018] System 100 also includes communication paths that enable communication between cache 120 and memory controller 130, communication between SOC fabric 150 and memory controller 130, and inter-chip communication paths that enable communication between memory controller 130 and memory devices 140. In some implementations, SOC 102 can conserve power by powering down one or more of the communication paths. Alternatively or additionally, SOC 102 can power down memory devices 140 to further conserve power. As another example, SOC 102 can enter a clock shut-down mode in which respective clock circuits are powered down for one or more devices.

[0019] Cache 120 is disposed in one of the data paths between SOC fabric 150 and memory controller 130. Thus, requests from client devices 110a-n to read from or write to memory device 140 pass through cache 120 or pass directly to memory controller 130. For example, client 110a may request to read from memory device 140, and the request is passed to cache 120 via SOC fabric 150. Cache 120 may process the request before forwarding the request to memory controller 130 for memory device 140. Alternatively or additionally, client 110a may make a request to read from memory device 140, and the request passes directly to memory controller 130 via SOC fabric 150, thus bypassing cache 120.

[0020] The cache 120 may cache read requests, write requests, or both from the client devices 110a-n. The cache 120 may cache read requests from the client devices 110a-n by responding to the request with data stored in the cache rather than fetching the data from the memory device 140. Similarly, the cache 120 may cache write requests from the client devices 110a-n by writing new data to the cache rather than writing the new data to the memory device 140. The cache 120 may later perform a write-back to write the updated data to the memory device 140.

[0021] The cache 120 may have dedicated cache memory, which may be implemented using dedicated registers or high-speed random access memory. The cache 120 may implement a caching policy that assigns different partitions, e.g., portions, or ways, of the cache memory, to different respective computing tasks. Thus, memory requests belonging to the same task may be serviced using the same allocated portion of the cache memory. For example, the SOC 102 in FIG. 1 illustrates the cache 120 as having several allocated partitions 122a-n. In general, the size (i.e., space in memory) and number of the cache partitions 122a-n may be predefined and / or dynamically adjusted to accommodate memory requests. In some cases, a subset of the partitions may be predefined to reserve space in the cache memory, while the remaining memory may be dynamically adjustable.

[0022] In some implementations, multiple tasks may be assigned the same partition of cache memory. To assign one or more tasks to a partition, cache 120 may examine the stream ID of the memory requests to determine which memory requests belong to the same task.

[0023] One example of these techniques includes assigning different partitions of the cache to different computing tasks executing on the same client device. For example, the cache 120 may examine the stream IDs of incoming memory requests to determine that some of the requests are associated with a process owned by a first task 112a and that some other requests are associated with a process owned by a second task 112b. Thus, to prevent these two tasks from competing with each other for cache resources, the cache 120 may assign a first partition 122a of the cache to the first task 112a executing on the client device 110a and a second partition 122b of the cache to the second task 112b executing on the same client device 110a. Alternatively or additionally, the first task 112a and / or the second task 112b executing on the client device 110a may not be assigned a partition and may therefore bypass the cache 120.

[0024] The cache 120 can also deallocate computing tasks from a partition or swap tasks from a partition to a different partition.

[0025] Another example includes allocating different partitions 112a-n of cache 120 to different buffers. For example, if SOC 102 is a GPU, each client device may perform a different function in the graphics processing pipeline. Thus, different data streams may be identified for render buffers, texture buffers, and vertex buffers, to name just a few.

[0026] Cache 120 may use a controller pipeline to process memory requests from SOC fabric 150. The controller pipeline executes cache logic to determine whether data is present in cache 120 or whether data needs to be fetched from or written to memory. Thus, the controller pipeline may also provide transactions to memory controller 130 when access to memory is required, such as a cache miss.

[0027] FIG. 2 is a diagram of an example subsystem 200. The subsystem 200 includes a candidate pool 210 that designates a particular set of computing tasks 212a-n based on their respective stream IDs. All tasks 222a-n that are not in the candidate pool 210 are non-candidates 220, which may or may not have stream IDs. Although not shown in FIG. 2, the candidate pool 210 may also include stream IDs for different threads of different computing tasks. The candidate pool 210 may include any number of stream IDs (e.g., no stream IDs, one stream ID, two stream IDs, etc.) corresponding to any number of tasks. In general, the candidate pool 210 designates a subset of computing tasks from the full set of computing tasks 112a-n executing on the client devices 110a-n, while the non-candidates 220 correspond to any computing tasks or other memory requests served by the client devices 110a-n that are not in the candidate pool 210.

[0028] 2 shows candidate pool 210 designated by SOC fabric 150. However, any suitable thread or processing device may designate candidate pool 210.

[0029] Candidate pool 210 can be modified in various ways based on various criteria. For example, SOC fabric 150 can be configured to add or remove stream IDs from candidate pool 210 based on cache 120 or main memory metrics. SOC fabric 150 can also add pre-specified stream IDs to candidate pool 210 at boot time. In some implementations, candidate pool 210 can be modified periodically or by a suitable algorithm that runs after certain conditions are met.

[0030] Subsystem 200 illustrates candidate pool 210, which communicates with cache 120 and memory controller 130. As previously described, cache 120 can be partitioned into several partitions 122a-n. In this case, only client devices executing computing tasks 212a-n designated by candidate pool 210 can source memory requests to cache 120, while all other tasks 212a-n source memory requests directly to memory controller 130. Thus, SOC fabric 150 can use candidate pool 210 to isolate specific computing tasks 212a-n for optimal utilization of cache 120. For example, tasks that are frequently executed but have predictable and / or limited data usage may be ideal for allocation to cache 120. While SOC fabric 150 can add these types of tasks to candidate pool 210, in general, candidate pool 210 can include any computing task.

[0031] An allocation engine of cache 120 can be configured to assign computing tasks to partitions 122a-n of cache 120 using the stream IDs. For example, the allocation engine can assign a first partition 112a of the cache to a memory request having a first stream ID and a second partition 112b of the cache to a memory request having a second stream ID. Thus, cache 120 can identify different computing tasks and assign different partitions of cache memory to each task based on their respective stream IDs. The allocation engine can use dedicated hardware circuitry of cache 120 to perform the allocation techniques described below.

[0032] Alternatively or additionally, the allocation process can be implemented in software, with the allocation engine having the host device's CPU execute the allocation algorithm. In some implementations, the allocation process can be executed by a dedicated thread or processing device. The processing device can be integrated within the SOC 102 or within the SOC fabric 150 along with multiple client devices 110a-n.

[0033] 3 is a flowchart of an example process 300 for allocating partitions of a cache. The example process 300 may be performed by one or more components of a cache. The example process 300 is described as being performed by a cache allocation engine on a SOC, suitably programmed in accordance with this specification.

[0034] The allocation engine identifies 310 stream IDs from a candidate pool of stream IDs corresponding to the computing task. As described above, memory requests belonging to a particular task are assigned an associated stream ID. In the exemplary process 300, the allocation engine identifies only stream IDs from the candidate pool for allocation.

[0035] Several different events can trigger the cache to initiate the allocation process 300 by identifying stream IDs for memory requests. For example, the cache can initiate allocation at boot time. As another example, the SOC can be configured to automatically generate a repartitioning trigger event when the SOC detects a change in execution or usage. The triggering event can be a signal or data received through the system indicating that the candidate pool has changed and that partitions of the cache need to be reallocated. Alternatively or additionally, the cache can identify stream IDs for memory requests by monitoring memory traffic. For example, the cache can maintain hit metric statistics for all partitions and assign partitions to stream IDs that meet certain criteria. Figure 5, described in detail below, is an exemplary process for adaptively allocating stream IDs by monitoring memory traffic and modifying the candidate pool.

[0036] A memory request may be associated with multiple stream IDs corresponding to multiple computing tasks, in which case the cache may repeat (at least in part) the exemplary process for each identified stream ID.

[0037] The allocation engine allocates (320) a partition of the cache to the memory request having the stream ID. The allocation engine may allocate any suitable partition of the cache, such as one or more lines, sets, ways, or some combination thereof. In some implementations, the partitions are allocated exclusively so that only memory requests having the specified stream ID can use the allocated cache resources.

[0038] The allocation process can distinguish between different types of computing tasks based on the stream ID. For example, the allocation engine can distinguish between tasks representing instructions and tasks representing data and allocate one partition of the cache to the instructions and another portion of the cache to the data. Furthermore, the allocation engine can distinguish between a first computing task performed by a client device and a second computing task performed by the same or a different client device and allocate different partitions of the cache to the different computing tasks. Considering a GPU as an example, the allocation process 300 can identify the stream IDs of a texture mapping thread and a polygon rendering thread and allocate different partitions to each respective thread.

[0039] In some implementations, the allocation engine may give special priority to tasks with stream IDs that store particular types of data structures and allocate different amounts of cache resources to each. For example, one data buffer that has a substantial impact on caching utilization is a page table. Thus, the allocation engine may treat data buffers that store page table data differently than buffers that store other types of data. For example, the allocation engine may allocate 1 MB of cache memory for page table pages and 4 kB of cache memory for other types of data buffers.

[0040] The cache then serves memory requests from client devices on the SOC based on the stream ID of the request (330). In doing so, the cache can effectively dedicate partitions of the cache to different computing tasks.

[0041] 4 is a flowchart of an example process 400 for servicing memory requests using a partition of a cache dedicated to a computing task. The example process may be performed by one or more components of a cache. The example process 400 is described as being performed by a cache on a SOC, for example, cache 120 of FIG. 1.

[0042] The cache receives a memory request 410. The memory request may be generated by a particular client device performing a particular computing task.

[0043] The cache identifies the stream ID of the task associated with the memory request 420. The stream ID may belong to a candidate pool and therefore may have a dedicated partition.

[0044] The cache determines whether the stream ID has a dedicated cache partition (430). In response to determining that the stream ID has a dedicated cache partition, the cache serves the memory request by using the dedicated cache partition (440). Otherwise, the cache serves the memory request using a default caching policy (450).

[0045] For example, a memory request for a GPU texture mapping thread may be received (410) by the cache and an identified stream ID (420). After determining that the stream ID has a dedicated cache partition (430), the cache may service the memory request using the partition (440). Similarly, a memory request for a GPU polygon rendering thread may be received (410) and the respective stream ID may be identified by the cache (420). The cache may determine that the stream ID does not have a dedicated partition (430), and therefore may service the memory request using a default caching policy (450).

[0046] 5 is a flowchart of an example process 500 for adaptively allocating computing tasks to partitions of a cache. The example process 500 may be performed by one or more components of the cache and a dedicated thread or processing device configured to perform operations by executing instructions. The processing device may be a client device on the SOC or may be separately integrated into the SOC. For example, the processing device may be a CPU. In some implementations, the operations of the processing device are performed by the cache.

[0047] The cache assigns computing tasks to cache partitions from the candidate pool (510). As described above with respect to FIG. 3, the cache can assign tasks from the candidate pool using their respective stream IDs. Generally, a subset of tasks from the candidate pool is assigned to a partition. The cache can assign tasks in various ways. The cache can also be instructed by a processing device to assign partitions. For example, the cache can be instructed by a processing unit to randomly assign partitions based on priorities, algorithms, etc. Multiple computing tasks can also be assigned to the same partition. Cache partitions can be predefined according to some desired memory configuration, typically based on the total amount of memory available in the cache. In principle, partitions can also be dynamically adjusted during the exemplary process 500, although this requires thorough cache invalidation.

[0048] The processing device monitors 520 the hit rates of all partitions of the cache. In the exemplary process 500, the processing device performs actions based on the hit metrics of the cache. However, the processing device may also monitor other performance metrics of the cache, such as cache size, associativity, replacement policy, etc.

[0049] For each partition, the processing device identifies 521 a hit rate for the partition. The processing device may continuously obtain or calculate per-partition hit rates for all partitions to identify a particular hit rate. The per-partition hit rate is the total number of cache hits divided by the sum of the cache hits and cache misses for a particular partition. In general, process 500 aims to maximize the hit rate for all partitions.

[0050] The processing device determines whether the partition's hit rate is lower than the partition's eviction threshold (522). The eviction threshold specifies a hit rate threshold and may be a measure of the desired baseline performance of the partition. Note that the eviction threshold may be different for different partitions. The eviction threshold may also vary depending on the particular implementation to tune the performance of the cache. For example, the eviction threshold may be programmed, specified, or dynamically created by a caching policy by cache 120 or SOC fabric 150.

[0051] If the hit rate is greater than the eviction threshold, the processing device continues to monitor the hit rate across all partitions (branch to 520). Thus, the cache maintains task assignments until the processing device determines that the hit rate is below the eviction threshold on a particular partition.

[0052] If the processing device determines that the hit rate is below the partition's eviction threshold, the cache deallocates the poorly performing task from the partition 523. In some cases, for example, when multiple tasks are assigned to a partition, the cache may deallocate multiple tasks from the partition.

[0053] The processing device determines whether the candidate pool is empty (524). Generally, process 500 continues indefinitely until the candidate pool is empty, at which point the processing device can perform further operations.

[0054] In some implementations, if the processing device determines that the candidate pool is empty, it issues an interrupt and / or a warning message (530). The warning message may alert the user with further instructions or may be issued without notification. Alternatively or additionally, the candidate pool may then be repopulated, for example, by the processing device, based on user command or a suitable algorithm.

[0055] On the other hand, if the candidate pool is not empty, the cache assigns a new task from the candidate pool to the partition (525). The cache can select a new task from the candidate pool in many ways. For example, the cache can be instructed by the processing device to select a new task randomly or based on an algorithm. The algorithm can be priority-based round-robin, FIFO (first-in-first-out), etc.

[0056] After determining that the hit rate for the partition is less than the eviction threshold, the processing device determines whether the hit rate is also less than the reclaim threshold for the partition (526). The reclaim threshold specifies a minimum value for the hit rate for the partition. Again, the reclaim threshold may be different for each partition. Like the eviction threshold, the reclaim threshold may vary depending on the particular implementation to tune the performance of the cache.

[0057] If the processing device determines that the hit rate is below the reclamation threshold for the partition, the processing device removes the deallocated computing task from the candidate pool (527). Thus, the removed task cannot allocate a partition in the cache when process 500 is repeated. That is, after removing the deallocated task from the candidate pool, the processing device continues to monitor the hit rate across all partitions (520).

[0058] On the other hand, if the hit rate is greater than the eviction threshold, the deallocated task remains in the candidate pool and the processing device continues to monitor the hit rate across all partitions (branch to 520). Thus, the deallocated task can be reassigned in a subsequent iteration of process 500.

[0059] Consider the stream IDs of the texture mapping thread and the polygon rendering thread in a candidate pool, taking the threads of a GPU as an example. At the start of the adaptive allocation process 500, the texture mapping thread may be assigned a partition of the cache, but the polygon rendering thread may not (510). While monitoring the hit metric (520), the process 500 may identify (521) the hit rate of the texture mapping allocation partition and determine (522) that the hit rate is below an eviction threshold. The process 500 may then deallocate (523) the texture mapping thread from the partition. After determining (524) that the candidate pool is not empty, the process 500 may assign (525) the polygon rendering thread to the partition. The hit rate of the texture mapping allocation partition may be below the eviction threshold, but the hit rate may be greater than the reclamation threshold (526). In this case, the texture mapping thread remains in the candidate pool, and the process 500 repeats by continuing to monitor (520) the hit rate. Thus, the overall effect of this iteration of process 500 was to swap the texture mapping thread with the polygon mapping thread for the poorly performing partition.

[0060] In general, the adaptive allocation process 500 self-tunes for an optimal configuration of computing tasks to be assigned to cache partitions, removing tasks from the candidate pool that tend to perform poorly.

[0061] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, or computer hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions can be encoded into an artificially generated transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus.

[0062] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or multiple computers. An apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0063] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, for example, files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0064] To say that one or more computer systems are configured to perform a particular operation or process means that the systems have installed thereon software, firmware, hardware, or a combination thereof that causes the operating system to perform the operation or process. To say that one or more computer programs are configured to perform a particular operation or process means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or process.

[0065] As used herein, "engine" or "software engine" refers to a hardware- or software-implemented input / output system that provides an output distinct from an input. An engine may be implemented in dedicated digital circuitry or as computer-readable instructions executed by a computing device. Each engine may be implemented in any suitable type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device, that includes one or more processing modules and computer-readable media. Furthermore, two or more of the engines may be implemented in the same computing device or different computing devices.

[0066] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry (e.g., FPGAs or ASICs), or a combination of special purpose logic circuitry and one or more programmed computers.

[0067] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic elements of a computer are a central processing unit for implementing and executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated into, special-purpose logic circuitry. Typically, a computer also includes, or is operatively coupled to receive and / or transfer data from, one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks. However, such devices are not required for a computer. Furthermore, a computer can be embedded in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0068] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all types of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0069] To interact with a user, embodiments of the subject matter described herein can be implemented in a host device. The host device can have a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, presence-sensing display, or other surface, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, voice input, or tactile input. Furthermore, a computer can interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer may also interact with a user by sending text messages or other types of messages to a personal device (e.g., a smartphone running a messaging application) and then receiving a reply message from the user.

[0070] In addition to the above embodiments, the following embodiments are also innovative. Embodiment 1 is a system, The system includes a plurality of integrated client devices, each configured to generate a memory request, each memory request having a respective pre-assigned stream ID representing a type of computing task to which the memory request belongs, the system further comprising: a cache configured to cache memory requests in a memory for each of the plurality of integrated client devices; The system is such that the cache has multiple partitions, and the cache is configured to assign a different partition to each memory request according to a stream ID of the memory request.

[0071] Embodiment 2 is the system of embodiment 1, in which memory requests belonging to different types of computing tasks have different stream IDs.

[0072] A third embodiment is the system according to any one of the first and second embodiments, wherein the cache is configured not to assign a partition to a specific stream ID.

[0073] Embodiment 4 is a system described in any one of embodiments 1 to 3, wherein the cache is configured to swap stream IDs from those using the first partition to those using the second partition.

[0074] A fifth embodiment is the system according to any one of the first to fourth embodiments, wherein the cache is configured to assign a plurality of different stream IDs for using the same partition.

[0075]

[0023] Embodiment 6 further includes a processing device configured to execute instructions to perform operations, the operations including: providing instructions to the cache for assigning partitions to stream IDs from a candidate pool of stream IDs; calculating a per-partition cache hit metric for each partition; and providing to the cache an instruction to change partition assignments of one or more stream IDs.

[0076] Embodiment 7 is a seventh embodiment, wherein calculating the per-partition cache hit metric includes calculating a hit ratio, and the operations include: determining that the hit rate of a partition is less than an eviction threshold; 7. The system of embodiment 6, further comprising: in response, deallocating one or more stream IDs from the partition and assigning new stream IDs to the partition from the candidate pool.

[0077] Embodiment 8 is a method for manufacturing a computer-readable recording medium, wherein the operation is: determining that the hit rate for the partition is less than a reclaim threshold; 8. The system of embodiment 7, further comprising: in response to this, removing the deallocated one or more stream IDs from the candidate pool.

[0078]

[0016] Embodiment 9 is a method for configuring a new stream ID from the candidate pool, comprising: random, Round Robin, First in, first out, or The system according to any one of embodiments 6 to 8, configured to allocate to the partitions using a selection algorithm based on any one of the priorities.

[0079] A tenth embodiment is the system according to any one of the seventh to ninth embodiments, wherein the eviction thresholds for at least some of the partitions are different.

[0080] An eleventh embodiment is the system according to any one of the eighth to tenth embodiments, wherein the regeneration thresholds for at least some of the partitions are different.

[0081] Embodiment 12 is a method performed by a device, the device comprising: a plurality of integrated client devices, each client device configured to generate memory requests, each memory request having a respective pre-assigned stream ID representing a type of computing task to which the memory request belongs, the device further comprising: a cache having a plurality of partitions; The method comprises: caching, by the cache, the memory request in a memory of each of the plurality of integrated client devices; and assigning, by the cache, a different partition to each memory request according to a stream ID of the memory request.

[0082]

[0023] Embodiment 13 is the method of embodiment 12, wherein memory requests belonging to different types of computing tasks have different stream IDs.

[0083]

[0033] Embodiment 14 is the method according to any one of embodiments 12 to 13, wherein the cache is configured not to assign partitions to specific stream IDs.

[0084]

[0047] Embodiment 15 is the method according to any one of embodiments 12 to 14, further comprising: swapping a stream ID from one using the first partition to one using the second partition.

[0085]

[0047] Embodiment 16 is the method according to any one of embodiments 12 to 15, further comprising allocating a plurality of different stream IDs for using the same partition.

[0086] Embodiment 17 is providing instructions to the cache for assigning partitions to stream IDs from a candidate pool of stream IDs; calculating a per-partition cache hit metric for each partition; 17. The method according to any one of embodiments 12-16, further comprising providing to the cache an instruction to change the partition assignment of one or more stream IDs.

[0087] Embodiment 18 is an embodiment of the present invention wherein calculating the per-partition cache hit metric includes calculating a hit ratio, and wherein the operation: determining that the hit rate of a partition is less than an eviction threshold; 18. The method of embodiment 17, further comprising: in response, deallocating one or more stream IDs from the partition and allocating new stream IDs to the partition from the candidate pool.

[0088] Embodiment 19 is determining that the hit rate for the partition is less than a reclaim threshold; 19. The method of embodiment 18, further comprising: in response, removing the deallocated one or more stream IDs from the candidate pool.

[0089]

[0033] Embodiment 20 further comprises: selecting a new stream ID from the candidate pool; random, Round Robin, First in, first out, or 20. The method according to any one of embodiments 17 to 19, further comprising allocating to the partition using a selection algorithm based on any one of the priorities.

[0090] Embodiment 21 is the method of any one of embodiments 18 to 20, wherein the eviction thresholds for at least some of the partitions are different.

[0091] Embodiment 22 is the method according to any one of embodiments 19 to 21, wherein the regeneration thresholds for at least some of the partitions are different.

[0092] Embodiment 23 is a computer storage medium encoded with a computer program, the program comprising instructions operable, when executed by a data processing device, to cause the data processing device to perform the method described in any one of embodiments 12 to 22.

[0093] While the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of patentable subject matter, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features of the invention that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to that subcombination or variations thereof.

[0094] Similarly, while operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence shown, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0095] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the operations recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A system comprising: The system includes a plurality of integrated client devices, each client device configured to generate memory requests, each memory request having a respective pre-assigned stream ID representing a type of computing task to which the memory request belongs, the system further comprising: a cache configured to cache memory requests in a memory for each of the plurality of integrated client devices; The system, wherein the cache has multiple partitions, and the cache is configured to assign a different partition to each memory request according to a stream ID of the memory request.

2. The system of claim 1 , wherein memory requests belonging to different types of computing tasks have different stream IDs.

3. The system of any one of claims 1 to 2, wherein the cache is configured to not assign partitions to specific stream IDs.

4. The system of any one of claims 1 to 3, wherein the cache is configured to swap stream IDs from those using a first partition to those using a second partition.

5. The system of any one of claims 1 to 4, wherein the cache is configured to allocate multiple different stream IDs for use with the same partition.

6. and a processing device configured to execute instructions for performing a plurality of operations, the plurality of operations comprising: providing instructions to the cache for assigning partitions to stream IDs from a candidate pool of stream IDs; calculating a per-partition cache hit metric for each partition; and providing instructions to the cache to change the partition assignment of one or more stream IDs.

7. Calculating the cache hit metrics for each partition includes calculating a hit ratio, and the operations include: determining that the hit rate of a partition is less than an eviction threshold; 7. The system of claim 6, further comprising: in response, deallocating one or more stream IDs from the partition and assigning new stream IDs to the partition from the candidate pool.

8. The plurality of operations include: determining that the hit rate for the partition is less than a reclaim threshold; and in response, removing the deallocated one or more stream IDs from the candidate pool.

9. The system selects a new stream ID from the candidate pool by: random, Round Robin, First in, first out, or A system according to any one of claims 6 to 8, configured to allocate to said partitions using a selection algorithm based on any one of the priorities.

10. The system of any one of claims 7 to 9, wherein the eviction thresholds for at least some of the partitions are different.

11. The system of any one of claims 8 to 10, wherein the regeneration thresholds for at least some of the partitions are different.

12. 1. A method performed by a device, the device comprising: a plurality of integrated client devices, each client device configured to generate memory requests, each memory request having a respective pre-assigned stream ID representing a type of computing task to which the memory request belongs, the device further comprising: a cache having a plurality of partitions; The method comprises: caching, by the cache, the memory request in a memory of each of the plurality of integrated client devices; and assigning, by the cache, a different partition to each memory request according to a stream ID of the memory request.

13. The method of claim 12 , wherein memory requests belonging to different types of computing tasks have different stream IDs.

14. The method of any one of claims 12 to 13, wherein the cache is configured to not assign partitions to specific stream IDs.

15. The method of any one of claims 12 to 14, further comprising swapping a stream ID from one using a first partition to one using a second partition.

16. The method of any one of claims 12 to 15, further comprising assigning multiple different stream IDs to use the same partition.

17. providing instructions to the cache for assigning partitions to stream IDs from a candidate pool of stream IDs; calculating a per-partition cache hit metric for each partition; The method of any one of claims 12 to 16, further comprising: providing instructions to the cache to change the partition assignment of one or more stream IDs.

18. Calculating the cache hit metrics for each partition includes calculating a hit ratio, and the operations include: determining that the hit rate of a partition is less than an eviction threshold; 20. The method of claim 17, further comprising: in response, deallocating one or more stream IDs from the partition and assigning new stream IDs to the partition from the candidate pool.

19. determining that the hit rate for the partition is less than a reclaim threshold; 20. The method of claim 18, further comprising: in response thereto, removing the deallocated one or more stream IDs from the candidate pool.

20. A new stream ID from the candidate pool random, Round Robin, First in, first out, or The method of any one of claims 17 to 19, further comprising allocating to said partitions using a selection algorithm based on any one of the priorities.

21. The method of any one of claims 18 to 20, wherein the eviction thresholds for at least some of the partitions are different.

22. The method of any one of claims 19 to 21, wherein the regeneration thresholds for at least some of the partitions are different.

Images