Software-directed hardware prefetcher for nested streams

Abstract

A system is provided, comprising: a memory for storing data; at least one cache for storing data copied from the memory to the cache; a processor for executing software code and reading data from the cache or the memory based on the software code; and a prefetcher for generating prefetch requests to prefetch data from the memory and storing copies of the prefetched data in the cache. The prefetcher is configured to, when processing a data stream generated by compiling the software code, perform the following operations: based on the processing of the data stream, identify scalable vector processing instructions in the software code executed by the processor; based on the scalable vector processing instructions, identify a plurality of data elements processed by the processor; generate one or more prefetch requests for prefetching a plurality of data elements corresponding to the identified plurality of data elements from the memory, and for storing copies of the prefetched plurality of data elements in the cache.

Description

Technical Field

[0001] This invention relates to computer processing, and more particularly to a software-guided hardware prefetcher for nested streams. Background Technology

[0002] In modern computer systems, retrieving data from main memory takes longer than processing the retrieved data. For example, retrieving data might take 150 nanoseconds, but processing it might only take a few nanoseconds. Random access time for retrieving data from main memory typically cannot keep pace with the increasing speed of processors. If data retrieval and processing are performed serially, the processor will be underutilized, spending most of its time waiting for data.

[0003] The standard technique for addressing main memory access time issues is to add memory caches. This involves adding a faster, smaller memory hierarchy between the processor and main memory. The processor hardware holds temporary copies of data or instructions in these caches. The processor can load its most recently stored data from the cache without waiting for the data to be loaded from main memory. While this works well for large amounts of data requiring processing, it is generally inefficient for data streams where each data element requires only minimal processing. This problem is particularly pronounced in high-performance systems capable of processing data elements very quickly.

[0004] For algorithms that use data streams, a software prefetcher or hardware prefetcher with memory caching is a common approach. A copy of the data is fetched from main memory and stored in one of the cache levels before the processor needs it. When the processor needs the data, the loading process is very fast. The data is prefetched into the memory cache.

[0005] High-performance Central Processing Units (CPUs) typically use hardware-based prefetchers. These devices use a trained algorithm that analyzes the processor's data retrieval patterns and uses this training algorithm to predict which memory locations will be used in the future. The hardware prefetcher then prefetches data from these memory locations and stores the data in the cache system. For this reason, hardware prefetchers are very efficient for direct streams densely packed in main memory. However, the training algorithm can be affected by irregular memory patterns generated by nested loops with main and indirect streams. In this case, the trained processor can prefetch incorrect data and degrade performance.

[0006] Software prefetchers use special prefetch instructions that are inserted into the code by application-level software or the compiler. These instructions directly trigger cache fill events. Without a mainstream, software-based prefetchers are efficient for direct streams and nested loops. However, they perform poorly for indirect streams or dependent nested loops. In these scenarios, the software prefetcher must fetch data from memory into the processor and perform computations to obtain the memory addresses of these streams. Memory address computations are performed twice, once for the prefetcher and once for the normal program flow. This can degrade software performance. Software prefetchers are most efficient if they are optimized for the hardware parameters of the caching system (e.g., cache line size). However, when software is optimized for the system's hardware parameters, it loses the flexibility to perform well on other hardware systems. Summary of the Invention

[0007] According to a first aspect of the present invention, a system is provided, comprising: a memory for storing data; at least one cache for storing data copied from the memory to the cache; a processor for executing software code and reading data from the cache or the memory based on the software code; and a prefetcher for generating prefetch requests to prefetch data from the memory and storing copies of the prefetched data in the cache, wherein the prefetcher is configured to, when processing a data stream generated by compiling the software code, perform the following operations: identifying scalable vector processing instructions in the software code executed by the processor based on the processing of the data stream; identifying a plurality of data elements processed by the processor based on the scalable vector processing instructions; generating one or more prefetch requests for prefetching a plurality of data elements corresponding to the identified plurality of data elements from the memory, and for storing copies of the prefetched plurality of data elements in the cache.

[0008] Therefore, the prefetcher according to embodiments of the present invention can process scalable vector instructions, wherein the software is unaware of how many vector elements are processed for each instruction. This can support prefetching in high-performance computing technologies.

[0009] The data stream may include at least one inner loop nested within at least one outer loop.

[0010] The prefetcher can generate the one or more prefetch requests by one or more of the following: a compiler for compiling the software code; the software code.

[0011] The prefetcher can also be used to: determine that a prefetch request to be processed has been discarded; and regenerate the discarded prefetch request.

[0012] The prefetcher can also be used to: determine that a first prefetch request and a second prefetch request are mapped to a common cache line in the cache, wherein the second prefetch request targets data that is not in the main stream of the data stream and not in the parent stream; and in response to determining that the first prefetch request and the second prefetch request are mapped to the common cache line, discard the second prefetch request. Therefore, the prefetcher can recognize that the completion of the first prefetch request implies that the second prefetch request is also completed.

[0013] The prefetcher can also be used to: determine that a first prefetch request and a second prefetch request are mapped to a common cache line in the cache, wherein the first prefetch request is for data in the main stream or parent stream of the data stream; and in response to determining that the first prefetch request and the second prefetch request are mapped to the common cache line, delay the execution of the second prefetch request.

[0014] The prefetcher can also be used to: delay the execution of the second prefetch request until each data element prefetched by the first prefetch request has been processed in the main stream or the parent stream. Therefore, the prefetcher can recognize that the second prefetch request can only be completed after the first prefetch request has been completed.

[0015] The prefetcher can also be used to: determine that data copied from the memory to the cache has been evicted; generate a prefetch request for prefetching the evicted data from the memory and storing a copy of the evicted data in the cache.

[0016] The prefetcher can also be used to: for each of one or more prefetch requests: determine whether the prefetch request is for data in the inner loop or the outer loop; and assign a priority to the prefetch request based on the determination. Therefore, different prefetches can be executed in different orders according to the assigned priorities.

[0017] The prefetcher can also be used to: execute one or more prefetch requests based on each assigned priority.

[0018] The prefetcher can also be used to: determine whether there is a risk that the prefetcher will generate a prefetch request for data that the processor does not yet need or no longer needs; and based on the determination, control the prefetcher's processing of the data stream. Therefore, the prefetcher can ensure that a prefetch request is generated before the software needs the data. This also allows the prefetcher to recover from situations where there is a risk that the prefetcher will fall behind the main program flow.

[0019] The prefetcher can be used to determine whether there is a risk of generating a prefetch request for data that the processor does not yet need or no longer needs by comparing the sensing stream associated with the processor with the sensing stream associated with the prefetcher based on the data stream.

[0020] The prefetcher can also be used to: determine that the prefetcher's processing of the data stream occurs before the processor executes the software code; and, in response to the determination, pause the prefetcher's processing of one or more loops in the data stream. This prevents the prefetcher's processing from occurring too far ahead of the software processing.

[0021] The prefetcher can also be used to control the processing of the data stream by delaying the generation of one or more prefetch requests.

[0022] The prefetcher can also be used to control the processing of the data stream by discarding one or more pending prefetch requests, or skipping the execution of one or more prefetch requests that have not yet started.

[0023] The one or more prefetch requests may be for data that the processor does not currently need when executing the software code based on the processor.

[0024] The one or more prefetch requests may be directed to data in one or more non-main loops of the data stream.

[0025] The prefetcher is also used to control the processing of the data stream by skipping one or more iterations of one or more loops in the data stream.

[0026] The one or more loops may correspond to instructions that have been executed by the processor.

[0027] The at least one cache may include a first cache and a second cache. The prefetcher may be used to control the processing of the data stream by modifying a pending prefetch request for the first cache such that the pending prefetch request is directed to the second cache.

[0028] The data stored in the memory may include one or more instructions. Therefore, the prefetcher can issue a prefetch request for data and instructions to be executed on the data.

[0029] According to another aspect of the present invention, a computer implementation method for prefetching data from memory using a prefetcher is provided, comprising: identifying scalable vector processing instructions in software code executed by a processor based on a data stream being processed by the prefetcher, wherein the data stream is generated by compiling the software code; identifying a plurality of data elements processed by the processor based on the scalable vector processing instructions using the prefetcher; generating one or more prefetch requests using the prefetcher; prefetching a plurality of data elements corresponding to the identified plurality of data elements from the memory using the one or more prefetch requests, and storing copies of the prefetched plurality of data elements in a cache for retrieval by the processor.

[0030] The scope of this invention is not necessarily limited to all aspects. Other aspects, features, and advantages will become apparent to those skilled in the art upon review of the following detailed description of embodiments. Attached Figure Description

[0031] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings, wherein:

[0032] Figure 1 A schematic diagram of a software-guided hardware prefetcher provided for an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a software boot hardware prefetcher used with a high-performance computing (HPC) server CPU, provided for embodiments of the present invention;

[0034] Figure 3 A schematic diagram of a software boot hardware prefetcher used with a server CPU, provided for an embodiment of the present invention;

[0035] Figure 4 Provided for embodiments of the present invention and Figure 2 A schematic diagram of the L3 cache operating together with other components in the context of the architecture shown;

[0036] Figure 5 A schematic diagram of a software-booted hardware prefetcher for use with a graphics processing unit (GPU) system (e.g., a graphics card or GPU server) provided for embodiments of the present invention;

[0037] Figure 6 This is a schematic diagram of a software-guided hardware prefetcher used with a Neural Processing Unit (NPU) system, provided as an embodiment of the present invention. Detailed Implementation

[0038] This invention relates to a software-booted hardware prefetcher for nested streams. While various embodiments of the invention are described below, the invention is not limited to these embodiments, and variations of these embodiments are likely to fall within the scope of the invention as defined only by the appended claims.

[0039] Generally, the following definitions may be applied to certain terms used throughout the invention:

[0040] Main memory: A medium-term storage device used to store instructions or data, such as Dynamic Random-Access Memory (DRAM). Main memory is randomly accessed, and each location is specified by a memory address that can be directly used by software instructions.

[0041] Cache: A short-term storage device used to temporarily store data and instructions used by software programs. Cache has a smaller capacity than main memory but is faster; however, the processor cannot directly access data locations in the cache. The cache contains temporary images of instructions or data stored in main memory and is referenced by software at the same memory address. These temporary images in the cache are maintained by hardware. Typically, computer systems have a cache hierarchy, with the fastest and smallest cache closest to the processor, and the largest and slowest cache closest to main memory.

[0042] Loading: Processor operations that read data from main memory.

[0043] Storage: Processor operations that write data to main memory.

[0044] A stream is a repeating sequence of data elements used by a software algorithm. Typically, software executes a loop and performs the same computational operation on each data element in the stream. Streams are either loaded from or stored in main memory. High-level software will assume that the data elements are processed serially, but compilers can use vector instructions to process stream elements in parallel.

[0045] Inductive flow: A loop counter used to control other types of flow. Inductive flows are generated by the processor but are not stored in main memory.

[0046] Direct stream: A stream of data elements whose memory address can be calculated using only the inductive stream or a constant. The inductive stream is called the parent stream of the direct stream.

[0047] Indirect stream: A stream of data elements for which the memory address of each data element needs to be obtained from a direct stream or an indirect stream. Indirect streams can also use constants. The parent stream of an indirect stream is a direct or indirect stream that provides the memory data elements used in address calculations. Each data element in an indirect stream requires a data element from its parent stream.

[0048] Dependency stream: direct stream or indirect stream.

[0049] Nested loops: A hierarchical control structure in which one loop (outer loop) contains another loop (inner loop). For each iteration of the outer loop, all iterations of the inner loop are executed. There can be any number of hierarchical levels in a nested loop. An outer loop can contain an inner loop, but an inner loop can also contain another inner loop. The perceptual flow of the outer loop is called the main flow, and the perceptual flow of the inner loop is called the subordinate flow.

[0050] Main dependency stream: The dependent data stream contained in the outer loop of a nested pair, used as a constant to control the inner loop. Constants can include, but are not limited to, the initial value, final value, or increment value of the dependent stream, or constants used for address calculation of the dependent stream.

[0051] Subordinate flow: A flow that has a dominant induction flow or dependency flow.

[0052] Parent stream: A stream that has child streams or is dependent on them. Each data element of the parent stream is used to generate the memory address of the child stream data element.

[0053] Child stream: A stream that has a parent stream as a dependency.

[0054] Example Streams: The following C code illustrates examples of each stream type.

[0055]

[0056]

[0057] Inductive flow: i, j

[0058] Direct flow: B, D, F, H, L, M

[0059] Indirect flow: N

[0060] Mainstream: i, B, D, F, H, L

[0061] Subordinate flows: j, M, N

[0062] Compiler: A software program that translates software written in a high-level language (such as C++) into low-level machine code instructions that are executed by a processor.

[0063] Processor: A hardware device that executes software instructions. In this article, "processor" can refer to a CPU designed for general-purpose computing, a GPU designed for graphics processing, or an NPU designed for machine learning applications. The distinguishing feature is that a processor executes software instructions stored in memory and uses these instructions to process data, which can be stored in memory or internally within the processor.

[0064] Vector instructions: A vector instruction is a single instruction that performs the same computational operation on multiple data items. Each item is called an element of the vector. Typically, compilers use vector instructions to execute multiple iterations of a loop simultaneously.

[0065] Scalable vector instructions: Vector instructions that are independent of the number of data elements in the vector. The processor hardware knows the number of elements in the vector, but the software instructions do not. The instructions only indicate what computational operation should be performed on each data element in the vector.

[0066] According to a first embodiment of the present invention, in Figure 1 A schematic diagram of a software-booted hardware prefetcher 100 is shown. The prefetcher 100 includes the following logic units and performs the following functions:

[0067] 1. A mechanism for programming the prefetcher 100 using nested stream configuration parameters and the interaction between these parameters. Before the software begins processing the stream, the prefetcher 100 stores this information in the nested stream configuration block 40. Software commands used to program the prefetcher 100 may include, but are not limited to, custom CPU instructions, custom CPU registers, or device memory registers.

[0068] 2. A mechanism that binds the relative instruction addresses seen by the compiler to the absolute memory addresses used by the hardware. This allows the prefetcher 100 to monitor the progress of stream processing via software.

[0069] 3. Prefetch generator block 30 automatically generates prefetches for all streams in all loops without subsequent software intervention until the loop completes. Prefetch generator 30 creates the same memory fetch pattern that will be used by the software. Prefetch generator 30 contains a counter equivalent to the sensing stream in the software. The counter uses the same initial value, final value, and increment value as the instruction stream in the software. Prefetcher 100 increments the counter before software intervention to ensure that prefetch generator 30 issues a prefetch request before the software needs the stream data.

[0070] 4. Pure software or software-booted hardware prefetchers can cause congestion in memory caching systems (i.e., main memory and cache) by issuing numerous prefetches mapped to the same hardware cache line. Caches handle much larger volumes of memory than software instructions can process. For example, a cache line might be 64 or 128 bytes wide, but software might handle data elements that are 4 or 8 bytes wide. Therefore, software-only or software-booted hardware prefetchers may generate redundant cache fill requests unless optimized for the cache line widths supported by the caching system hardware.

[0071] Therefore, prefetch request and trace block 60 is used to request prefetching from the memory caching system and track the status of incomplete prefetch requests. Prefetch request and trace block 60 also performs the following operations:

[0072] a. If the memory caching system has already discarded the prefetch, then request a new prefetch.

[0073] b. Detect the times when different prefetch requests map to the same cache line and merge them into the same prefetch request for the memory caching system.

[0074] c. If the memory caching system has evicted the previous prefetch and moved the data to a different cache level, then request a new prefetch.

[0075] 5. A mechanism for prefetcher 100 to automatically load data from a memory cache after prefetching is complete. This includes tracking scenarios where multiple prefetch requests are merged into a single memory cache prefetch, but the cache prefetch contains master or parent data from multiple streams. Multiple data elements can be loaded from a single cache request.

[0076] 6. Hardware buffer 50 is used to store the main and parent data before they are needed from the memory cache. This prevents the outer loop from blocking the inner loop. Multiple data elements from each stream can be stored in the buffer.

[0077] 7. A hardware mechanism for reconfiguring internal streams using buffered master data when needed. Master data changes the loop parameters configured in the software. This can be achieved through several mechanisms, such as, but not limited to, directly changing the nested stream configuration or by maintaining a separate current stream parameter table.

[0078] 8. Hardware mechanism for generating indirect prefetches using buffered parent data. The prefetch generator uses parent stream data stored in a hardware buffer. The parent data is used to generate indirect prefetches but is not part of the stream configuration. Each prefetch in the indirect stream requires a new parent data value. This mechanism may or may not support reordering of data elements in the indirect stream.

[0079] 9. As mentioned above, pure software or software boot prefetchers often struggle to handle scalable vector processing instructions that are independent of the hardware vector length. The hardware vector length indicates the number of data elements processed per instruction. Instructions can be Single Instruction, Multiple Data (SIMD). Software writing is independent of the CPU's hardware capacity. High-level software is written in loops that process one element at a time, but the compiler generates instructions that process one vector at a time. Processors (e.g., Figure 2 The processor 200)

[0080] It handles vectors that simultaneously represent multiple loop iterations. Without scalable vector processing instructions, the software must be manually optimized for each hardware platform.

[0081] Therefore, a hardware-based program monitor 10 is used to track the execution of instructions in each loop via software. The program monitor 10 can automatically detect scalable vector processing instructions and measure the number of vector elements processed by each instruction. The program monitor 10 maps physical vector processing instructions to software-independent loop parameters, such as sensed variables.

[0082] 10. A mechanism in prefetcher 100 distinguishes between main loop prefetching and secondary loop prefetching, and assigns a priority to each prefetch. Prefetch priorities can trigger (but are not limited to) selective discarding or prefetch reordering.

[0083] 11. As mentioned above, pure hardware prefetchers use trained algorithms to predict the next needed data, but these algorithms may err and fetch data that was never needed. Pure software or software-booted hardware prefetchers do not err, but they may lag behind the main program flow. If a pure software or software-booted hardware prefetcher lags behind the program flow, the prefetcher may request a cache to fill memory locations already processed by the main program.

[0084] This can cause congestion in the memory caching system and reduce overall performance.

[0085] Therefore, the adaptive prefetch rate control block 20 controls the prefetch request rate to ensure that prefetching is requested before the software needs the data. Adaptive prefetch rate control 20 prevents prefetch requests from being issued prematurely in the future and can recover when the software progress catches up with the prefetcher. Adaptive prefetch rate control can also recover when the software progress has exceeded the prefetcher's. Flow control mechanisms may include, but are not limited to:

[0086] a. Pause the prefetch generator 30 while waiting for the processor to catch up.

[0087] b. Pause the generation of prefetches for individual streams.

[0088] c. Discard prefetches that are no longer needed because the software progress has exceeded the prefetcher 100.

[0089] d. Skip loop iterations already executed by the software.

[0090] e. When the prefetcher 100 is not lagging behind, but is at risk of lagging behind the software, actively skip the loop iteration.

[0091] f. If prefetcher 100 is at risk of falling behind the software, then actively discard non-primary prefetches.

[0092] g. Change the target cache level for prefetching.

[0093] 12. A method for resuming prefetching after a process thread content switching save and restore cycle. This mechanism can be used, but is not limited to, supporting software reading and writing the internal state of prefetcher 100. Software can freeze prefetcher 100 and read its internal state and configuration. Later, the software can restore prefetcher 100 to its previous state and resume execution from the point of interruption. This mechanism works correctly even if another software thread is using prefetcher 100 between the save and restore events.

[0094] Go to Figure 2 This illustrates the use of a prefetcher 100 in a high-performance computing (HPC) server CPU according to an embodiment of the present invention.

[0095] In this example, there are two processors 200, each with its own L1 cache 210, L2 cache 220, and prefetcher 100. The two processors 200 share an L3 cache 230, which is directly connected to the system's main memory 240. Main memory 240 can be internal memory on the same integrated circuit as the processors 200, or an external memory cache system such as Double Data Rate (DDR) DRAM. This is an illustrative example. In real server systems, there are typically more processors, or even multiple L3 caches. L1 or L2 caches may or may not be shared. Since prefetcher 100 is only connected to processor 200 and L1 cache 210, the exact hierarchy of caches beyond L1 and main memory is not important. If prefetcher 100 prefetches L2 cache 220 to reduce congestion of L1 cache 210, prefetcher 100 needs to know that an L2 cache exists, but its exact configuration is not important.

[0096] The cache contains temporary copies of the data located in main memory 240. Each cache represents a subset of main memory 240. In server CPU systems, a consistent grouping cache is used. Typically, the cache is configured such that the cache closest to processor 200 is smaller and faster. Caches closer to main memory 240 are typically larger and slower, with main memory 240 having the largest capacity and slowest speed.

[0097] Each processor 200 corresponds to a prefetcher 100. Each prefetcher 100 is connected to the processor 200 and the L1 cache 210. The prefetcher 100 is programmed by software, and each prefetcher 100 initiates sending prefetch requests to the L1 cache 210. Each prefetcher 100 loads data from the L1 cache 210 as needed to process the main and indirect streams. The prefetcher 100 monitors the software flow within the processor 200 and performs flow control operations as needed.

[0098] See Figure 3 According to one embodiment of the present invention, the prefetcher 100 can also be used in a conventional cloud server CPU or a dedicated high-performance server CPU.

[0099] The server CPU comprises numerous L3 clusters arranged in a ring. Each L3 cluster contains an L3 memory cache 330 and several processor clusters 350. The L3 clusters are connected in a ring structure, with main memory blocks 340 inserted at various points in the ring. When the L3 cache 330 needs to access main memory 340, the request is routed through each L3 cluster 330 until it reaches the correct main memory 340. Alternatively, the server CPUs can be connected in a star configuration of the L3 clusters. There can be fewer or more cache levels. As long as at least one cache level exists, the prefetcher 100 will operate through any memory cache configuration.

[0100] The L3 cluster consists of the following components: Figure 4 As shown in the diagram, as described above, each L3 cluster contains several processor clusters 350. Each processor cluster 350 contains an L2 memory cache 320, and one or more processors 300 have an L1 memory cache 310. Each processor 300 corresponds to a prefetcher 100. The prefetcher 100 is connected to both the processor 300 and the L1 cache 310. The instruction cache is not considered in this diagram because the prefetcher 100 is only connected to the data cache closest to the processor 300. A system can have shared caches for instructions and data, or it can have separate instruction and data caches. Different tiers may also employ different schemes (e.g., separate L1 caches for instructions and data, and combined L2 caches containing both instructions and data).

[0101] A standard coherent memory caching scheme is used. Processor 300 issues load and store instructions to L1 cache 310, which either loads data from main memory 340 or stores new data in main memory 340. If the memory location used for the load operation is not contained in L1 cache 310, a data fill request is triggered to copy the data into L1 cache 310. The data fill request propagates to all cache levels until a copy of the data is found. For example, if the data is found in L2 cache 320, L2 cache 320 returns the data to L1 cache 310. If the data is not found in L2 cache 320, the request is propagated to L3 cache 330 and even main memory 340 to retrieve the data. Storing to main memory 340 uses the same process. Typically, the data written in a data storage operation is less than a complete cache line. During a store operation, the memory caching system first executes a cache fill request to load the cache line into L1 cache 310. After the cache line is written to L1 cache 310, store operations continue to overwrite parts of the cache line. The modified cache line is returned to main memory 340.

[0102] Prefetcher 100 also issues data fill requests to L1 cache 310 and loads data from L1 cache 310. If prefetcher 100 issues a fill request to L1 cache 310 and the data is already in L1 cache 310, no further action is required. Prefetcher 100 does not load data for every stream type. Prefetcher 100 only needs to load data for the main and parent streams of indirect streams. In an alternative implementation, prefetcher 100 maintains a list of data locations stored in L1 cache 310. In this implementation, prefetcher 100 will only send cache fill requests for memory locations that are not yet stored in L1 cache 310.

[0103] Each prefetcher 100 has three interfaces connected to its processor 300. The first interface (configuration) is used by the software to program the configuration of nested loop flows and their interactions. The second interface (monitoring) allows the prefetcher 100 to monitor which instructions the processor 300 executes. The prefetcher 100 uses the monitor interface to track the progress of the software and determine when flow control events are needed. The third interface (read) is a data read interface used by the processor 300 to read the configuration and internal state data of the prefetcher 100. This interface is used to save the internal state of the prefetcher 100 during context switch save and restore cycles. The first interface (configuration) is also used to restore the internal state of the prefetcher 100 during content switch restore events.

[0104] The following describes the general process in a server application.

[0105] 1. The software running on processor 300 configures prefetcher 100 using stream parameters and their linkages. Once complete, processor 300 starts prefetcher 100 and begins executing nested loops.

[0106] 2. Prefetcher 100 begins processing the stream and generates fill requests for L1 cache 310. The logic in prefetch generator 30 within prefetcher 100 simulates the nested loop structure of the main software program. Prefetcher 100 maintains a counter that simulates the sensed flow used to control the loop. Prefetcher 100 increments the simulated sensed flow using the same initial, increment, and final values ​​used by the software. Although prefetcher 100 processes the same stream as the software, each step is performed first by prefetcher 100.

[0107] 3. L1 cache 310 processes data filling requests and, when necessary, propagates the requests to other cache levels or main memory 340. If a request is propagated to a cache level higher than L1 cache 310, the data filling is eventually returned to L1 cache 310 through the cache hierarchy.

[0108] 4. Prefetcher 100 loads data from the main stream and dependent streams with indirect child streams from L1 cache 310 for further processing. Prefetcher 100 monitors the data returned to L1 cache 310 to know when the data is available in L1 cache 310. If the stream is not the main stream or parent stream, prefetcher 100 does not need to track the time it takes to complete the fetch request.

[0109] 5. The prefetcher 100 monitors the instructions executed by the software and performs flow control operations as needed. Flow control operations prevent the prefetcher 100 from progressing too far ahead of the software or enable the prefetcher 100 to recover from falling behind the software progress.

[0110] Streams #2 through #5 are continuous and interleaved in different iterations of the nested loop streams. Whenever the sensing stream counter changes, all dependent streams generate direct prefetch requests using the new value. Whenever a prefetch request for a dependent parent stream completes, prefetcher 100 loads the parent data into buffer 50. When the parent data is available in buffer 50, indirect streams generate prefetch requests. Whenever the inner loop sensing stream reaches its final value, the outer loop sensing stream moves to the next value. When all mainstream values ​​are loaded into buffer 50, the inner stream configuration parameters are updated.

[0111] To improve performance, prefetcher 100 can request master values ​​before the inner loop needs them and store these master values ​​in buffer 50. When the inner loop completes, a new master value is retrieved from buffer 50 and used by the next iteration of the inner loop. When the master value is removed from buffer 50, the outer loop can request a new set of master values. Master values ​​can iterate several times before the inner loop uses the value. This prevents the inner loop from being interrupted while waiting for the outer loop.

[0112] Because of the different latency of cache and memory caching systems, memory caching systems may reorder cache fill requests. The order of operations for inductive and direct stream prefetch requests is exactly the same as the order in the software flow. Indirect streams may be emitted in roughly the same order as the program flow. Indirect streams depend on loading data from dependent streams, but memory caching systems may return cache fill requests in an order different from the order of requests. Flow control mechanisms will ensure that these streams roughly conform to the program order.

[0113] The preceding description has assumed that processor 300 is single-threaded and runs only one software program at a time. High-performance processors can execute multiple threads (typically two) concurrently. Prefetcher 100 supports multi-threaded processors and has one of the following two options:

[0114] 1. Each running thread contains a different prefetcher block. To prevent security vulnerabilities, each thread can only see one prefetcher block.

[0115] 2. The prefetcher 100 contains a distinct nested stream configuration table 40 for each thread. The prefetch generator 30 processes each thread either serially or in parallel. To prevent security risks, each thread can only see one nested stream configuration table 40 and can only read the internal state of its own set of streams.

[0116] Figure 5 An embodiment of the present invention illustrates a prefetcher 100 for GPU systems such as graphics cards or GPU servers. Mobile phones typically use small GPUs directly within their processor chips. GPU systems are designed for wide-vector computations and have very large memory bandwidth. They are well-suited for graphics processing algorithms but can also be used for machine learning algorithms.

[0117] The GPU comprises an array 560 of small processor units (PRs) 500. Processor array 560 acts as a vector processor, with each PR 500 processing one element of a vector. Instruction unit 570 broadcasts the same instruction to each PR 500 in array 560 that processes Single Instruction Multiple Data (SIMD) vector computation. Processor array 560 acts as a scalable vector processor, and the number of PR units 500 in array 560 determines the maximum vector length. Each processor array 560 in the diagram has an L1 memory cache 510. Clusters 550 of processor arrays 560 share an L2 cache 520. Topology parameters (e.g., the number of PR units per array or the number of processor arrays per cluster 550) are not critical. Figure 5 The topology shown is for illustrative purposes only. Furthermore, in Figure 5 In this configuration, multiple processor clusters 550 share the same main memory 540. Different implementations may also have more or fewer cache levels. The exact topology of the memory hierarchy is not important, as long as at least one cache level exists, each prefetcher 100 can operate.

[0118] In GPU applications, the operation of prefetcher 100 is essentially the same as in server CPU applications. One prefetcher 100 corresponds to one processor array 560. Prefetcher 100 is configured by high-level software and monitors instruction unit 570 to perform prefetch flow control. Prefetcher 100 issues prefetch commands and loads data into L1 cache 510. For dependent streams without child streams, prefetching can be requested to populate L1 cache 510 or caches at different levels. Main streams or dependent streams that are parent streams of other streams must be fetched into L1 cache 510 and loaded for further processing. The operation of prefetcher 100 is the same as in server applications.

[0119] Because GPUs support very wide vectors, vector support features in prefetcher 100 are important. GPUs can also support very wide cache lines. For wide cache lines, the ability of prefetcher 100 to merge prefetches into a single cache line to fill the request is also important. Traditional graphics applications work well with dense matrix computations, but many newer Artificial Intelligence (AI) or Machine Learning (ML) applications use sparse matrix computations. This makes prefetcher 100 very useful when GPUs are used for AI and ML applications.

[0120] For example:

[0121]

[0122] In this example, A, C, and D will all use the same cache line. C and D are the main cache lines controlling the inner loop, but A is not. A is prefetched first, and then a cache fill request is sent. C and D wait for A because they need the same memory fill request that A has already sent. In this sense, the memory fill requests of A, C, and D have been merged.

[0123] When data is written to the cache, nothing is done for A; only C and D need to copy data from the cache. Copy B[j+1] for C and B[j+2] for D.

[0124] Figure 6 An embodiment of a prefetcher 100 in a Neural Processing Unit (NPU) system is illustrated. The NPU system is optimized for Artificial Intelligence (AI) and Machine Learning (ML) applications and can be used on large server systems or handheld mobile devices.

[0125] The NPU 620 is a dedicated processor for performing matrix computations or other computations common in machine learning algorithms. It comprises an array 610 of processors 600 optimized to perform a small number of computations, such as multiply-and-accumulate (MAC) operations. The NPU 620 performs many computations per cycle. This requires loading large amounts of data from main memory 640 and caching it in a data buffer 660. The NPU 620 is similar to a GPU but supports only a limited number of instructions and computation types. It should be noted that this example is an illustrative example of an NPU 620 that only supports MAC operations. Specific NPU implementations may support more complex instructions, but the exact number and type are not important.

[0126] The MAC array 610 serves as a vector processor, and the data buffer 660 acts as a single-level cache. Traditional NPUs have a Direct Memory Access (DMA) engine that uses very high data rates to fill the data buffer 660 with data elements from the primary memory 640. Typical NPU DMA engines perform very well when processing dense matrices, but struggle with sparse matrices and the irregular data patterns they generate.

[0127] If the NPU can handle sparse matrices, prefetcher 100 acts as an accelerator to prefill data buffer 660 before it is needed. For this, NPU 620 requires complex instructions to extract sparse data patterns and fill data buffer 660 with the correct data. If NPU 620 does not contain the correct complex instructions to extract sparse data, prefetcher 100 can be used as a DMA engine for NPU 620. In this embodiment, the operation of prefetcher 100 differs slightly from the CPU embodiment. When prefetcher 100 executes as a DMA engine, processor array 610 waits for prefetcher 100 to fill the required data before computation begins. Processor array 610 monitors the progress of prefetcher 100 to ensure data is ready. This means that prefetcher 100 never needs to use flow control to stay ahead of the software's progress. Prefetcher 100 will extract all data without skipping or discarding data stream elements. Prefetcher 100 will use flow control to prevent data from getting too far ahead of the software's progress, but never behind. The control logic of the NPU 620 will need to check the internal state of the prefetcher 100 to know when the data is ready.

[0128] Prefetcher 100 can also be used to extract both dense and sparse matrices. Prefetcher 100 can extract dense matrices at the same rate as a typical DMA engine, but it performs significantly better than a DMA engine when extracting sparse matrices. Dense matrices will have no indirect flow.

[0129] The NPU embodiment can also be implemented using a Field Programmable Gate Array (FPGA). In the FPGA embodiment, the MAC array 610, MAC controller 630, DMA controller 650, and prefetcher 100 will be implemented in the FPGA device. The data buffer 660 can be implemented in the FPGA's internal memory or in faster external memory. An advantage of the FPGA embodiment is that the MAC processor 600 can be customized to suit the application's requirements. The system can even use the MAC processor 600 instead of different types of processors performing different types of data processing. The FPGA embodiment can be reconfigured in different ways for each new software application loaded into the system.

[0130] Typically, the prefetcher according to embodiments of the present invention can process scalable vector instructions, wherein the software is unaware of how many vector elements are processed for each instruction. This can support prefetching of high-performance computing technologies such as CPUs with ARM SVE, CPUs with Intel AVX-512, graphics processing units, and neural processing units.

[0131] Furthermore, the prefetcher according to embodiments of the invention can use sophisticated adaptive control mechanisms to recover from situations where prefetching has fallen behind the software or is at risk of falling behind. This may include skipping loop iterations and discarding ongoing prefetching. This can support the prefetcher in prefetching only data beneficial to the software and prevent the prefetcher from degrading performance by congesting the memory caching system with unnecessary requests.

[0132] For example, consider the following example:

[0133]

[0134] Everything is controlled by counters (sensing flows) j and k. There are actually four counters: j_software, k_software, j_prefetcher, and k_prefetcher. The prefetcher counter is maintained internally within the prefetcher and is used to generate prefetches. The software counters are used by the actual software for data processing. The program monitor 10 measures the software counters by inspecting the program flow. For correct prefetching, the prefetcher must precede the software execution process, i.e., j_prefetcher > j_software and k_prefetcher > k_software.

[0135] The adaptive prefetch rate control 20 performs various flow control actions to maintain optimal ahead-of-run values, run_ahead_j = j_prefetcher – j_software and run_ahead_k = k_prefetcher – k_software.

[0136] If k_prefetcher < k_software, the prefetcher for stream k is already behind the software. Using the current value of k_prefetcher is meaningless. The data is no longer needed, nor are any prefetch requests for E and G. A new value of k_prefetcher must be selected to restore the prefetcher to an appropriate state, and ongoing prefetching for E and G should be discarded.

[0137] If j_prefetcher < j_software, the prefetcher is indeed behind. Both the inner loop and the outer loop are behind. In this case, everything the prefetcher is doing is no longer needed and should be discarded.

[0138] If run_ahead_k is less than 0, the prefetcher is already behind the software. If run_ahead_k is steadily decreasing and moving away from the optimal value, there is a risk that the prefetcher's progress will fall behind the software.

[0139] In addition, the prefetcher according to an embodiment of the present invention can assign different priorities to different types of streams based on their relationship with other streams in the nested loop hierarchy. This can avoid performance problems caused by row header congestion. In a nested loop, the latency in the main stream prefetch blocks multiple iterations of the inner loop. The latency of the inner loop stream that is not the parent stream does not congest other streams.

[0140] In addition, the prefetcher according to an embodiment of the present invention can merge prefetching to the same cache line into a single fill request and then split them into multiple data load operations if they are needed as the main stream or parent stream for generating other prefetching. This can minimize the number of fill requests in the memory cache hierarchy to reduce congestion and power. This also supports the software in configuring the prefetcher without knowing the underlying cache line size.

[0141] Furthermore, the prefetcher according to embodiments of the present invention can be used to prefetch data record structures stored in memory connected by linked lists. In the linked list data structure, each data structure contains a set of data fields. One of these fields is a pointer to the next data structure in the list. To track the linked list, the indirect stream must be its own parent stream. The indirect stream fetches data from memory containing the position of the next data record in the list. The indirect linked list stream requires a starting position, but then it continuously issues prefetches, loads the prefetched data, and calculates the next prefetch position in the list. The initial value can be configured by software or come from the main stream. Other data in the records are indirect streams with the linked list stream as their parent stream. A processor using this system can automatically traverse the linked list and prefetch the data contained in the list. Exemplary software for tracking the linked list is shown below.

[0142]

[0143] Streams B, C, and D contain the data fields associated with each record. Stream A is the memory address where each record resides.

[0144] Optionally, the indirect linked list stream can store the final value in the configuration or main stream. The stream is complete when the prefetched data equals the final value.

[0145] In addition, the prefetcher can be modified to perform instruction prefetching. Consider the following software code as an example:

[0146]

[0147] In this software, the value of data M determines which software function is used to process data N. In addition to prefetching all data elements, a modified prefetcher can be used to prefetch the correct instructions into the cache. It requires a new function table containing the base addresses of functions FunctionCall0 through FunctionCall4. The prefetch generator 30 checks which condition is true. Using the result of the condition check, the prefetch generator 30 generates instruction prefetches by looking up the correct addresses in the function table. The prefetcher can generate both data and instruction space prefetches.

[0148] When used in conjunction with the terms “comprising” or “including” in the claims and / or specification, unless expressly otherwise specified, the word “a” or “an” may mean “one”, but it also has the same meaning as “one or more,” “at least one,” and “more than one.” Similarly, unless expressly otherwise specified, the word “another” may mean at least a second or more.

[0149] As used in this article, references to “approximately”, “about”, or “basically” to a number mean that the number is within a range of + / - 10%.

[0150] Although the present invention has been described in conjunction with specific embodiments, it should be understood that the present invention is not limited to these embodiments, and those skilled in the art can make changes, modifications and variations to these embodiments without departing from the scope of the present invention.

[0151] It is also conceivable that any aspect or any part of an embodiment discussed in this specification may be implemented or combined with any other aspect or any part of an embodiment discussed in this specification.

Claims

1. A system, characterized in that, include: A memory used to store data; At least one cache is provided for storing data copied from the memory to the cache; A processor for executing software code and reading data from the cache or the memory based on the software code; A prefetcher is configured to generate a prefetch request to prefetch data from the memory and store a copy of the prefetched data in the cache. The prefetcher is configured to perform the following operations when processing the data stream generated by compiling the software code: Based on the processing of the data stream, identify the scalable vector processing instructions in the software code executed by the processor; Based on the scalable vector processing instructions, identify multiple data elements that the processor has processed; Generate one or more prefetch requests for prefetching multiple data elements corresponding to the identified multiple data elements from the memory, and for storing copies of the prefetched multiple data elements in the cache.

2. The system according to claim 1, characterized in that, The data stream includes at least one inner loop nested within at least one outer loop.

3. The system according to claim 1 or 2, characterized in that, The prefetcher generates the one or more prefetch requests by one or more of the following: A compiler used to compile the software code; The software code.

4. The system according to any one of claims 1 to 3, characterized in that, The pre-extractor is also used for: It has been determined that the pending prefetch request has been dropped; Regenerate the discarded prefetch request.

5. The system according to any one of claims 1 to 4, characterized in that, The pre-extractor is also used for: Determine the common cache line in the cache to which the first prefetch request and the second prefetch request are mapped, wherein the second prefetch request is for data that is not in the main stream of the data stream and is not in the parent stream; In response to determining that the first prefetch request and the second prefetch request are mapped to the common cache line, the second prefetch request is discarded.

6. The system according to any one of claims 1 to 5, characterized in that, The pre-extractor is also used for: Determine that the first prefetch request and the second prefetch request are mapped to a common cache line in the cache, wherein the first prefetch request is for data in the main stream or parent stream of the data stream; In response to determining that the first prefetch request and the second prefetch request are mapped to the common cache line, the execution of the second prefetch request is delayed.

7. The system according to claim 6, characterized in that, The pre-extractor is also used for: The execution of the second prefetch request is delayed until each data element prefetched by the first prefetch request is processed in the mainstream or the parent stream.

8. The system according to any one of claims 1 to 7, characterized in that, The pre-extractor is also used for: It has been determined that the data copied from the memory to the cache has been evicted; A prefetch request is generated to prefetch the evicted data from the memory and a copy of the evicted data is stored in the cache.

9. The system according to any one of claims 1 to 8, characterized in that, The pre-extractor is also used for: For each of one or more prefetch requests: Determine whether the prefetch request is for data in the inner loop or the outer loop; Based on the determination, a priority is assigned to the prefetch request.

10. The system according to claim 9, characterized in that, The pre-extractor is also used for: Based on the priority of each assignment, the one or more prefetch requests are executed.

11. The system according to any one of claims 1 to 10, characterized in that, The prefetcher is also used for: Determine whether the prefetcher risks generating a prefetch request for data that the processor does not yet need or no longer needs; Based on the determination, the prefetcher is controlled to process the data stream.

12. The system according to claim 11, characterized in that, The prefetcher is used to determine whether there is a risk of generating a prefetch request for data that the processor does not yet need or no longer needs: Based on the data stream, the sensing stream associated with the processor is compared with the sensing stream associated with the prefetcher.

13. The system according to claim 11 or 12, characterized in that, The pre-extractor is also used for: The processing of the data stream by the prefetcher is determined before the processor executes the software code; In response to the determination, the prefetcher suspends processing of one or more loops in the data stream.

14. The system according to any one of claims 11 to 13, characterized in that, The prefetcher is also used to control the processing of the data stream in the following ways: Delay the generation of one or more prefetch requests.

15. The system according to any one of claims 11 to 14, characterized in that, The prefetcher is also used to control the processing of the data stream in the following ways: Discard one or more pending prefetch requests; or Skip one or more prefetch requests that have not yet started execution.

16. The system according to claim 15, characterized in that, The one or more prefetch requests are for data that the processor does not currently need when executing the software code based on the processor.

17. The system according to claim 15 or 16, characterized in that, The one or more prefetch requests are for data in one or more non-main loops of the data stream.

18. The system according to any one of claims 11 to 17, characterized in that, The prefetcher is also used to control the processing of the data stream in the following ways: Skip one or more iterations of one or more loops in the data stream.

19. The system according to claim 18, characterized in that, The one or more loops correspond to instructions that have been executed by the processor.

20. The system according to any one of claims 11 to 19, characterized in that, The at least one cache includes a first cache and a second cache; The prefetcher is used to control the processing of the data stream in the following ways: Modify the pending prefetch request for the first cache so that the pending prefetch request is for the second cache.

21. The system according to any one of claims 11 to 20, characterized in that, The data stored in the memory includes one or more instructions.

22. A computer implementation method for prefetching data from memory using a prefetcher, characterized in that, include: Based on the data stream being processed by the prefetcher, the prefetcher is used to identify scalable vector processing instructions in software code executed by the processor, wherein the data stream is generated by compiling the software code; Based on the scalable vector processing instructions, the prefetcher is used to identify multiple data elements that the processor has processed; The prefetcher is used to generate one or more prefetch requests; The one or more prefetch requests are used to prefetch multiple data elements corresponding to the identified multiple data elements from the memory, and copies of the prefetched multiple data elements are stored in a cache for retrieval by the processor.

Images