Prefetching address translations in memory management unit

Abstract

This document describes technologies related to efficiently providing address translations with a memory management unit (MMU). The MMU prefetches address translations and stores the prefetched translations directly in the TLB by injecting prefetch requests into a stream of memory access requests. The MMU generates the prefetch requests using a programmable prefetching policy upon encountering a translation lookaside buffer miss and stores the prefetch requests in a prefetch buffer. In some examples, prior to adding a prefetch request to the prefetch buffer, the MMU compares the prefetch request against already buffered prefetch to requests to remove any overlapping regions. The MMU injects the buffered prefetch requests into the main traffic of memory access requests opportunistically using a defined arbitration scheme.

Description

PREFETCHING ADDRESS TRANSLATIONS IN MEMORY MANAGEMENT UNITBACKGROUND

[0001] Various computing systems use memory management units (MMUs) for the purpose of memory virtualization. MMUs are responsible for translating virtual addresses, which are generated by a processing unit into physical addresses within the memory of the computing system. MMUs are designed for efficient memory management and facilitate virtual memory and process isolation.

[0002] MMUs are designed to execute tiered page walk processes to translate virtual addresses to physical addresses. In a tiered page walk, the MMU traverses through multiple levels of page tables to translate a virtual address to a physical address. The traversal through multiple levels of page tables can result in additional latency for the address translation process. To mitigate this, MMUs are further equipped with translation lookaside buffers (TLBs) that cache translations for pages that have been accessed recently.SUMMARY

[0003] This document generally relates to systems, methods, and devices for prefetching address translations in a memory management unit (MMU) using a translation lookaside buffer (TLB) lookup-result based prefetching scheme.

[0004] Example aspects include technologies related to efficiently providing address translations with an MMU. In some implementations, the MMU prefetches address translations. The MMU stores the address translations directly in the TLB by injecting prefetch requests into a stream of memory access requests being translated by the MMU, in contrast to MMUs that store the prefetched translations in a separate walk cache. Prefetch requests are generated upon encountering a TLB miss using a programmable prefetching policy, which allows for more effective prefetching in nuanced memory access traffic patterns as compared to MMUs that are limited to forward linear prefetching. The MMU stores the generated prefetch requests in a prefetch buffer. In some implementations, prior to adding a prefetch request to the prefetch buffer, the MMU compares the prefetch request against already buffered prefetch to requests to remove any overlapping regions. In some implementations, the MMU injects the bufferedprefetch requests into the main traffic of memory access requests opportunistically using a defined arbitration scheme.

[0005] In one aspect a method is disclosed. The method includes receiving, at a memory management unit (MMU), a memory access request in a stream of memory access requests. The memory access request includes a virtual address and at least one non-address tag. The method further includes determining that a mapping for the virtual address is not cached in a translation lookaside buffer (TLB) of the MMU and generating a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB. Where generating the prefetch request includes retrieving a programmable prefetching policy from a look-up table based on the at least one non-address tag, determining one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request, and storing the prefetch request in a prefetch buffer. The method further includes injecting the prefetch request from the prefetch buffer into the stream of memory access requests to preload one or more address translations for the one or more pages in the TLB.

[0006] In another aspect a memory management unit (MMU) is disclosed. The memory management includes an address translation buffer unit including a translation lookaside buffer (TLB) and an address translation control unit configured to perform translation table walks, he MMU is configured to receive a memory access request in a stream of memory access requests, the memory access request including a virtual address and at least one non-address tag, determine, by the address translation buffer unit, that a mapping for the virtual address is not cached in the TLB, and generate a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB. Where to generate the prefetch request includes to retrieve a programmable prefetching policy from a look-up table based on the at least one non- address tag, determine one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request, and store the prefetch request in a prefetch buffer. The MMU is further configured to inject the prefetch request from the prefetch buffer into the stream of memory access requests and perform, by the address translation control unit, a translation table walk for the prefetch request to preload one or more address translations for the one or more pages in the TLB.

[0007] In yet another aspect, a computing system is disclosed. A computing system comprising a processing unit and memory management unit (MMU) comprising a translation lookaside buffer (TLB). The MMU is configured to perform address translations for the processing unit including to receive a memory access request in a stream of memory access requests from the processing unit, the memory access request including a virtual address and at least one non-address tag, determine that a mapping for the virtual address is not cached in the TLB, and generate a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB. Where to generate the prefetch request includes to retrieve a programmable prefetching policy from a look-up table based on the at least one non-address tag, determine one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request, and store the prefetch request in a prefetch buffer. Where the MMU is further configured to inject the prefetch request from the prefetch buffer into the stream of memory access requests to preload one or more address translations for the one or more pages in the TLB.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The prefetching scheme disclosed herein stores prefetched translations directly in the TLB to reduce the latency for accessing a prefetched translation for the first time by eliminating the need for the address translation buffer unit to retrieve the prefetched translation from a cache in the address translation control unit. The programmable prefetching policy allows for effective prefetching in nuanced memory access traffic patterns (e.g., non-linear traffic, backward addresses, various strides between pages, etc.) resulting in a higher TLB hit rate in various contexts. MMU instruction protocols have commands to prefetch translations but traditional initiators do not have prior knowledge of their upcoming traffic patterns and do not know the current state of TLB entries inside the MMU. Therefore, traditional MMU’s are not able to make use of prefetch and / or stash translation commands in an efficient way. The systems and methods disclosed herein can be used to identify patterns in the traffic to predict future traffic and only generate prefetch requests for TLB misses, thereby reducing the overall number of prefetch requests while simultaneously increasing the effectiveness of the prefetch requests. In this manner, the systems and methods disclosed improve performance and reduce latency of an MMU performing address translations. Additionally, improved TLB hit rate allows for power usage improvements byproviding more opportunities to clock gate the address translation control unit. Dynamically collapsing overlapping regions of prefetch requests in the prefetch buffer minimizes prefetcher traffic that may interfere with initiator traffic, thereby improving performance and reducing power requirements for the MMU.

[0009] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 is a schematic flow diagram illustrating a process for using a memory management unit to perform address translations.

[0011] FIG. 2 illustrates an example method for prefetching address translations.

[0012] FIG. 3 is a schematic flow diagram illustrating an example process for managing a prefetch buffer.

[0013] FIG. 4 illustrates an example method for determining one or more pages to include in a prefetch request.

[0014] FIG. 5 illustrates a schematic flow diagram illustrating a method for injecting a prefetch request into a stream of memory access requests.

[0015] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0016] This document generally relates to systems, methods, and devices for prefetching address translations in a memory management unit (MMU) using a translation lookaside buffer (TLB) lookup-result based prefetching scheme.

[0017] A MMU is used within a computing system to provide address translations quickly and efficiently. In some implementations, the MMU prefetches address translations and stores the address translations directly in the TLB by injecting prefetch requests into a stream of memory access requests being translated by the MMU. The MMU generates the prefetch requests upon encountering a translation lookaside buffer (TLB) miss. For example, the TLB miss may be indicative of a memory access request which jumped to a region of memory that is from theprevious recent memory access requests that are cached in the TLB (e.g., because a new application is now requesting memory access). Upon encountering a TLB miss, a prefetch request for upcoming pages is generated and stored in a prefetch buffer. Based on information in the memory access request that resulted in a TLB miss, the MMU determines pages that are likely to be accessed in incoming traffic. For example, the memory access request may include an identifier corresponding to a particular application that accesses memory in a pattern known by the MMU. The MMU stores the prefetched address translations directly in the TLB, thereby increasing the TLB hit rate for incoming traffic.

[0018] In some examples, the MMU stores the prefetch requests in a prefetch buffer. The overlapping prefetches are collapsed dynamically in a prefetch buffer of the MMU. In some examples, prior to adding a prefetch request to the prefetch buffer, the MMU compares the prefetch request against already buffered prefetch to requests to identify and remove any overlapping regions (e.g., areas in memory being prefetched by more than one prefetch request). Some examples include dynamically running an optimization on every incoming prefetch request against the already buffered prefetches where overlapping regions are removed from the incoming prefetch request using a power-efficient sieve.

[0019] In some examples, the buffered prefetch requests are injected into the main traffic of memory access requests opportunistically. In some examples, a software defined arbitration scheme is used to opportunistically inject prefetch requests from the prefetch buffer into the main traffic. By injecting the prefetch requests directly into the main traffic the address translations for the prefetch request will be stored directly in the TLB. For example, after the translation control unit completes a page table walk for the prefetch request.

[0020] In some examples, the MMU is implemented on a system on chip (SoC) integrated circuit for a mobile device. In the context of a mobile device (and other general purpose computing devices- laptops, desktops, tablets, smart watches, video game consoles, general purpose servers, etc.) it is common for the memory access requests to jump around to different areas in memory (. For example, because multiple applications may be running concurrently or near concurrently (e.g., multiple streams of traffic), including applications running in the background. In these examples, application identifiers are used to provide information to the MMU on the application that is requesting the translation, in order to allow the MMU to determine what pages (if any) should be prefetched for the requested application. The prefetching scheme disclosed hereinfurther results in performance and power savings which improve the performance usability of mobile devices.

[0021] FIG. 1 is a schematic flow diagram illustrating a process 100 for using a memory management unit (MMU) 101 to perform address translations. The MMU 101 includes an address translation buffer unit 104, an address translation control unit 106, and a translation generator 103. The MMU 101 receives address translation requests from an initiator 102. The MMU interfaces with the memory 122 to access a page table 124.

[0022] The MMU 101 is responsible for translating virtual addresses to physical addresses in a computing system. In the example shown, MMU 101 converts a virtual address of memory access request in the stream of memory access requests 130 generated by the initiator 102 into a physical address in memory. The translation for the physical address is provided with the memory access results 138. The MMU 101 enables features for virtual memory and process isolation that are critical for security and virtualization. In some examples, the MMU 101 is implemented on a system on chip (SoC) integrated circuit for a mobile device. In some examples, the MMU 101 is a system memory management unit (SMMU). In some implementations, the MMU tracks memory use in fixed-sized blocks referred to as pages. In some examples, the page size is 4 kilobytes (KB).

[0023] The initiator 102 is the entity that starts an operation which requires an address translation. Examples of the initiator 102 include a central processing unit (CPU), graphical processing unit (GPU), I / O device, or any other computational units that require address translations. Although only one initiator 102 is shown, the MMU 101 may be configured to provide memory address translations for multiple initiators.

[0024] The stream of memory access requests 130 can include one or more memory access requests to read and / or write to a virtual address. Each memory access request can include nonaddress tags. Examples of non-address tags include an identifier for an application that initiated the memory access request, a source of the request (e.g., the initiator 102), and other information related to the memory access request. The stream of memory access requests may come from multiple initiators including the initiator 102.

[0025] The memory access results 138 includes a translation of a virtual address to a physical address. A memory access result 138 is provided to the initiator, in response to the MMU 101 receiving a memory access request in the stream of memory access requests 130.

[0026] The translation generator 103 manages a stream of memory translation traffic including a stream of memory access requests 130 from the initiator 102. In the example shown, The translation generator 103 includes a prefetch buffer 110. The translation generator 103 receives a stream of memory access requests 130 from the initiator 102. In some examples, the translation generator 103 injects prefetch requests 134 from the prefetch buffer 110 into the stream of memory access requests 130. An example method for opportunistically injecting prefetch requests into the stream of memory access requests 130 is illustrated and described in reference to FIG. 5.

[0027] The translation generator 103 generates prefetch requests. In some implementations, the translation generator 103 determines pages that are likely to be accessed in upcoming memory access requests based on a translation buffer miss for a memory access request. The translation generator 103 looks at the miss traffic(e.g., the demand translation buffer miss 132) going from the address translation buffer unit 104 to the address translation control unit 106 and uses the information included in the miss traffic to determine what pages to prefetch. The translation buffer miss provides information on the current state of the TLB 120. For example, the TLB 120 does not have a cached translation for the virtual address in the memory access request. The memory access request can include other information using non-address tags. As discussed above, the non-address tags can include an identifier for an application that initiated the memory access request. The information in the non-address tags is used to determine a prefetching policy. The prefetching policy can include a number of pages to prefetch as well as a policy for determining which pages to prefetch relative to the page for the virtual address of the current memory access request. An example method for generating a prefetch request is illustrated and described in reference to FIG. 4.

[0028] The prefetch buffer 110 stores generated prefetch requests. In some examples, the prefetch buffer 110 is a first-in-first-out (FIFO) buffer. The entries in the prefetch buffer 110 are collapsed to remove overlapping regions. An example for how entries in the prefetch buffer 110 are collapsed to remove overlapping regions is illustrated and described in reference to FIG. 3. In some examples, the prefetch requests in the prefetch buffer 110 are injected to the main stream of traffic opportunistically using a defined arbitration scheme. An example for injecting prefetch requests into the main traffic is illustrated and described in reference to FIG 5.

[0029] The address translation buffer unit 104 includes a translation lookaside buffer (TLB) 120. In some examples, the TLB 120 is a cache that is configured to store address translations in TLB entries. The address translation buffer unit 104 is configured to receive a virtual address and searches for a matching translation in the TLB 120. If a matching translation is found, the MMU 101 outputs the translation with the memory access result 138. If a matching translation is not found, the MMU 101 sends an address translation request to the address translation control unit 106. Although only a single address translation buffer unit 104 is shown in FIG. 1 the MMU 101 can include multiple address translation buffer units, each having a corresponding TBL, including address translation buffer units responsible for managing translations for multiple initiators. In some examples, the prefetching requests are further determined on a per address translation buffer unit basis for the multiple address translation buffer units. For example, based on a context of the type or types of initiators that use the address translation buffer unit. Examples of the address translation buffer unit include translation buffer units (e.g., TBUs).

[0030] The TLB 120 includes a cache storing translations for recently accessed pages in TLB entries. The TLB 120 is implemented to have a high hit rate for incoming memory access requests on the cached translations in order to reduce latency. The TLB 120 stores address translations for prefetch requests in the same manner as those stored from memory access requests. For example, the address translation control unit 106 directly loads the translations for memory access requests (e.g., from demand translation buffer miss 132) and for prefetch requests (e.g., from the speculative translation buffer miss 136).

[0031] The demand translation buffer miss 132 occurs when an address translation for a virtual address in the memory access request is not cached in the TLB 120. The demand translation buffer miss 132 is sent to the address translation control unit 106 to initiate page table walks to retrieve the address translation. The demand translation buffer miss 132 is also sent to the translation generator 103 to determine whether a prefetch request should be generated.

[0032] The speculative translation buffer miss 136 is a translation buffer miss that results from a prefetch request. Because the prefetch requests are injected into the main stream of traffic by the translation generator 103, they may generate a speculative translation buffer miss 136 when the address translation buffer unit 104 attempts to perform the address translation for one or more pages of the prefetch request. The speculative translation buffer miss 136 is sent to the address translation control unit 106, where the address translation control unit 106 performs a table walkto identify the translations for the requested one or more pages. In some implementations, the translation generator 103 ignores speculative translation buffer misses 136 to avoid triggering the generation of prefetch requests based on an expected miss of a prefetch request.

[0033] The address translation control unit 106 controls and manages address translations including to perform a page table translation process. When the address translation control unit 106 receives an address translation request (e.g., as part of the demand translation buffer miss 132 or the speculative translation buffer miss 136), the address translation control unit 106 performs translation table walks by accessing the page table 124 in the memory 122 to search for the address translation. In some examples, the address translation control unit 106 includes one or more caches that are walked (e.g., before performing a walk of the page table 124) according to a priority to try to reduce the number of table walks that are performed. The address translation control unit 106 loads translations for requested pages in the TLB 120. In some examples, when there are no translation buffer misses (e.g., demand translation buffer miss 132 and speculative translation buffer miss 136), the clock for the address translation control unit 106 can be disabled to reduce power consumption. Examples of the address translation control unit include translation control units (e.g., TCUs, translation control, and / or translation controllers).

[0034] The memory 122 is a physical memory unit of a computing system. The memory 122 stores, among other data, a page table 124. The page table 124 is a data structure storing the mappings between virtual addresses and physical addresses. Each mapping is stored in the page table 124 as a page table entry. In some examples, each entry stores a physical address of a page in memory, whether the page is present, writable, accessible etc. In some examples, the page table 124 includes hierarchical page tables organized to allow for more efficient management and faster access to the page table entries.

[0035] In the example shown, the process 100 begins with the initiator 102 sending a memory access request in the stream of memory access requests 130. The memory access request includes a virtual address. The memory access request is received at the address translation buffer unit 104 which determines whether the TLB 120 includes an address translation for the virtual address of the memory access request. Upon the TLB 120 not including the address translation for the virtual address of the memory access request, the address translation buffer unit 104 sends a demand translation buffer miss 132 with the memory access request to the address translation control unit 106, where the address translation control unit 106 performs apage table walk to determine the address translation to return to the initiator 102 and stores the translation for the page in the TLB 120. The translation generator 103 also receives the demand translation buffer miss 132 with the memory access request. The translation generator 103 generates a prefetch request based on non-address tags and the virtual address of the memory access request. For example, using the method 400 illustrated and described in FIG. 4. The generated prefetch request is stored in the prefetch buffer 110. The translation generator 103 injects the prefetch request 134 into the stream of memory access requests 130. For example, using the process 500 illustrated in FIG. 5. When the prefetch request is received at the address translation buffer unit 104, the address translation buffer unit 104 determines whether the TLB 120 is storing address translations for the requested pages. If the TLB 120 does not include an address translation for a requested page it will generate a speculative translation buffer miss 136. The address translation control unit 106 receives the speculative translation buffer miss 136 from the address translation buffer unit 104. The address translation control unit performs a table walk of the page table 124 to determine an address translation for the missed page. The address translation control unit 106 loads the determined translation directly into the TLB.

[0036] In one non-limiting example, an initiator 102 switches to executing instructions for a new application that requires accessing an area of memory that has not recently been accessed. This switch to the new application will likely trigger a demand translation buffer miss 132 since the translation for the area of memory is unlikely to be cached in the TLB 120. The translation generator 103 processes the memory access request, including non-address tags of the memory access request to determine if a prefetch request should be generated (e.g., because of an indication that a certain page will likely be accessed in the near future by the application) and a prefetch criteria. The prefetch criteria may include a number of pages to prefetch, a direction of pages to prefetch, an interval between pages to prefetch, etc.

[0037] In one non-limiting example, the initiator 102 is a high bandwidth initiator that quickly exhausts all addresses in a given page by hitting the TLB 120. The translation generator 103 can be configured (either by software or hardware) to generate prefetch requests for more pages to reduce upcoming latency because the initiator 102 is a type that quickly exhausts translations on any given page.

[0038] In one non-limiting example, the initiator 102 accesses memory in a linear manner. When upon a demand translation buffer miss 132, the translation generator 103 will generate prefetchrequests for the pages linearly following the page of the virtual address of the current memory access request that missed. Because these pages are pre fetched and stored in the TLB 120, the address translation buffer unit 104 will not miss when the initiator exhausts the page of the virtual memory access request.

[0039] FIG. 1 illustrates an example architecture for performing the process 100. However, the techniques disclosed herein (including the process 100) can be performed on a system and / or MMU with a different architecture including different components than those shown in FIG. 1.

[0040] FIG. 2 illustrates an example method 200 for prefetching address translations. In some examples, the method 200 is performed by the MMU 101 illustrated and described in reference to FIG. 1. The method 200 includes the operations 202, 204, 206, and 208.

[0041] At the operation 202, the MMU receives a memory access request. The memory access request includes a virtual address for translation and at least one non-address tag. In some examples, the memory access request is a memory read request. In some examples, the memory access request is a memory write request. The at least one address tag can include an identifier for an application that requested the memory access request. In some examples, the at least one non-address tag includes an identifier for a source (e.g., a central processing unit (CPU), a graphical processing unit (GPU), or other processing units) of the memory access request.

[0042] At the operation 204, the MMU determines whether a mapping for the virtual address is cached in the TLB. If the mapping for the virtual address is cached in the TLB the translation is returned to a requesting initiator. If the mapping for the virtual address is not cached in the TLB, the address translation buffer unit will generate a translation buffer miss (e.g., the demand translation buffer miss 132 shown in FIG. 1).

[0043] At the operation 206, upon determining that the mapping for the virtual address is cached in the TLB, the MMU generates and stores a prefetch request in a prefetch buffer. In some examples, generating the prefetch request includes retrieving a programmable prefetching policy from a look-up table based on the non-address tags and determining one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request. In some examples, the MMU uses the programmable prefetching policy to identify pages that are likely to be accessed by upcoming memory access requests. For example, a page adjacent to the page containing the virtual address of the currentmemory access request, such as the next page in linear order. An example method for generating a prefetch request is illustrated and described in reference to FIG. 4.

[0044] The programmable prefetching policy can include an indication of whether this is the type of transaction where the MMU should prefetch (e.g., where prefetching is likely to be helpful), a number of pages to prefetch, a direction pages to prefetch relative to the page of the virtual address of the current memory access request (e.g., forward to backward from a currently being accessed page), an interval to jump between pages to prefetch (e.g., prefetch every other page, every third page, every tenth page, every 100thpage, etc.).

[0045] In some examples, storing the prefetch request in the prefetch buffer includes determining whether the prefetch request includes an overlapping region with a second prefetch request in the prefetch buffer and updating the prefetch request to remove the overlapping region. In some of these examples, the MMU uses a power efficient sieve to compare the first region with the second region. An example method for storing the prefetch request in the prefetch buffer is illustrated and described in reference to FIG. 3

[0046] At the operation 208, the MMU injects the prefetch request into a stream of memory access requests. In some examples, injecting the prefetch request into the stream of memory access requests further includes for a given cycle determining that the MMU received a valid memory translation request and that the prefetch buffer includes a valid prefetch request and injecting either the valid memory translation request or the prefetch request for the given cycle based on a defined arbitration scheme. In some examples, the defined arbitration scheme includes a weighted round robin. In some of these examples, weights for the weighted round robin are determined based on the programmable prefetching policy. An example method for injecting a prefetch request in to a stream of memory access requests is illustrated and described in reference to FIG. 5

[0047] FIG. 3 is a schematic flow diagram illustrating an example process 300 for managing a prefetch buffer 110. In some examples, the process 300 is performed by the translation generator 103 of the MMU 101, illustrated and described in reference to FIG. 1. In some examples, the MMU (e.g., the generator of the MMU) performs the process 300 as part of the operation 206 illustrated and described in reference to FIG. 2. The process 300 includes the operations 302, 304, 306, 308, 310, and 312.

[0048] In the example shown, the prefetch buffer entries are split between non-address tags 332 and address tags 334. The splitting of tag information allows the MMU to perform a power efficient comparison. For example, by eliminating the comparison of addresses of prefetch requests when there is not a match between the non-address tags of the prefetch requests.

[0049] In some examples, each incoming prefetch request is compared against already buffered prefetch requests to dynamically remove overlapping regions. Removing overlapping regions improves the performance of the MMU by reducing the overall number of addresses included with prefetch requests that are injected into the mainstream of memory access requests. In some examples, reviewing overlapping addresses in the prefetch buffer 110 reduces power consumption by removing redundant prefetch requests that are sent to the address translation buffer unit. In some examples, such a feature improves battery life of a device (e.g., a portable computing device).

[0050] At the operation 302, the MMU receives an incoming prefetch request. The incoming prefetch request includes non-address tags of the memory access request and one or more virtual addresses for one or more pages included in the prefetch request.

[0051] At the operation 304, the MMU determines whether the non-address tags of the incoming prefetch request match any of the non-address tags in the prefetch buffer 110. If none of the address tags match then the translation generator can determine that there are no overlapping regions between the incoming prefetch request and the prefetch requests stored in the prefetch buffer 110 and the prefetch request can be inserted at a head of the FIFO (at the operation 312). If there is a match between the incoming prefetch request and one or more buffered prefetch requests, the requested pages in the incoming prefetch requests are compared with requested pages in the one or more buffered prefetch requests.

[0052] At the operation 306, the MMU determines whether addresses of the incoming prefetch request overlaps with the one or more buffered prefetch requests that had matching non-address tags. In some examples, a power efficient sieve with two stages is used to determine if address regions are overlapped to save power.

[0053] At the operation 308, the MMU generator computes an overlap region. The overlap region includes areas in memory in the incoming prefetch request that are already set to be accessed by a buffered prefetch request.

[0054] At the operation 310, the MMU updates the incoming prefetch request. The incoming prefetch request is updated to remove the overlapping region from the incoming prefetch request At the operation 312, the MMU inserts the incoming prefetch request in the prefetch buffer. The MMU is thereby able to dynamically collapse the buffered prefetch requests as new prefetch requests are received.

[0055] FIG. 4 illustrates an example method 400 for determining one or more pages to include in a prefetch request. In some examples, the method 400 is performed by the translation generator 103 of the MMU 101, illustrated and described in reference to FIG. 1. In some examples, the MMU (e.g., the translation generator of the MMU) performs the process 300 as part of the operation 206 illustrated and described in reference to FIG. 2. The method 400 includes the operations 402, 404, and 406.

[0056] At the operation 402, the MMU processes at least one non-address tag of the memory access request. In some examples, the memory access request may include any number of nonaddress tags for example 1 tag, 2 tags, 5 tags, 16 tags, 32 tags, 64 tags, 128 tags, etc.

[0057] At the operation 404, the MMU retrieves a programmable prefetching policy based on the at least one non-address tag. In some examples, the programmable prefetching policy is retrieved from a look-up table (or data organization structure) using the non-address tags. For example, the non-address tags may define a context and a configuration for prefetching for the context may be retrieved from a lookup table (or data organization structure). The programmable prefetching policy is configurable to meet a purpose of a given initiator (e.g., a CPU, GPU, a particular GPU substrate, a tensor processing unit (TPU); an Al accelerator processing unit; or

[0058] application specific processing unit, or other processing units) application initiating the memory access requests, and combinations thereof. The programmable prefetching policy may be configured according to other factors related to the memory access request. In some examples, the programmable prefetching policy includes an identifier which is indicative of if this is the type of transaction, where prefetching should be used. The prefetching policy can also define a number of pages to prefetch, a direction to prefetch relative to a current page (or memory access request address), an interval to jump between prefetchers, etc. In some examples, the programmable prefetching policy is configurable by software. In some examples, hardware prefetching policy circuitry is used. For example, hardware logic that generates a certain prefetching request when the hardware logic detects a certain type of TLB miss.

[0059] At the operation 406, the MMU determines one or more pages to include the with the prefetch request based on the programmable prefetching policy. One or more prefetch requests are generated for the one or more pages.

[0060] FIG. 5 illustrates a schematic flow diagram illustrating a process 500 for injecting a prefetch request into a stream of memory access requests. In some examples, the process 500 is performed by the translation generator 103 of the MMU 101, illustrated and described in reference to FIG. 1. In some examples, the MMU (e.g., the translation generator of the MMU) performs the process 500 as part of the operation 208 illustrated and described in reference to FIG. 2. The process 500 includes operations 502, 504, 506, 308, 510, 512, and 514.

[0061] At the operation 502, the generator determines whether the stream of memory access requests includes an incoming valid memory access request from the initiator. If there is not a valid memory access request, the generator next determines whether the prefetch buffer includes a valid prefetch request at the operation 504. If there is a valid prefetch request, the generator can inject the prefetch request into the stream of memory access requests. If there is not a valid prefetch request, the MMU will not inject a prefetch request into the stream of memory access requests, at the operation 512.

[0062] If at the operation 502, the MMU determines that there is a valid access request, the generator will next determine whether the prefetch request is valid at the operation 510. If it is determined that the prefetch buffer does not include a valid prefetch request, the MMU will not inject a prefetch request into the stream of memory access requests, at the operation 512.

[0063] If at the operation 510 the MMU determines that the prefetch buffer does include a valid prefetch request, the MMU determines whether to inject the prefetch request according to a defined arbitration scheme. In some examples, the arbitration scheme is defined in software. In some examples, the arbitration scheme defines a weighted round robin to set a prioritization of memory access requests against prefetch requests. For example, the weights can be set such that 70% of the time that the memory access requests conflict with a prefetch request, the memory access request is prioritized. Other weightings can be used. In some examples, the weights are set on a per application basis. In other examples, the weights are set on a per address translation buffer unit basis. The address tag of the prefetch request (or of the multiple prefect requests in the prefetch buffer) may also be used to define a priority as compared to the memory access requests.

[0064] In addition to the embodiments of the attached claims and the embodiments described above, the following numbered embodiments are also innovative:

[0065] Embodiment 1 is a method comprising: receiving, at a memory management unit (MMU), a memory access request in a stream of memory access requests, the memory access request including a virtual address and at least one non-address tag; determining that a mapping for the virtual address is not cached in a translation lookaside buffer (TLB) of the MMU; generating a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB, wherein generating the prefetch request includes: retrieving a programmable prefetching policy from a look-up table based on the at least one non-address tag; determining one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request; and storing the prefetch request in a prefetch buffer; and injecting the prefetch request from the prefetch buffer into the stream of memory access requests to preload one or more address translations for the one or more pages in the TLB.

[0066] Embodiment 2 is the method of embodiment 1 , wherein the one or more pages are determined using the programmable prefetching policy to identify pages that are likely to be accessed by upcoming memory access requests based on the virtual address of the memory access request.

[0067] Embodiment 3 is the method of embodiment 1 or 2, wherein storing the prefetch request in the prefetch buffer further comprises: determining whether the prefetch request includes an overlapping region with a second prefetch request in the prefetch buffer; and updating the prefetch request to remove the overlapping region.

[0068] Embodiment 4 is the method of embodiment 3, wherein determining whether the prefetch request includes the overlapping region with the second prefetch request in the prefetch buffer further comprises: determining that the non-address tag of the memory access request is a match of a second non-address tag of the second prefetch request; and comparing a first region for the memory access request with a second region for the second prefetch request to determine the overlapping region.

[0069] Embodiment 5 is the method of claim 4, wherein the MMU uses a power efficient sieve to compare the first region with the second region.

[0070] Embodiment 6 is the method of any of embodiments 1 -5, wherein the at least one nonaddress tag includes an identifier for an application requesting the memory access request.

[0071] Embodiment 7 is the method of any of embodiments 1-6, wherein the at least one nonaddress tag includes an identifier for a source of the memory access request.

[0072] Embodiment 8 is the method of embodiment 7, wherein the source one of; a central processing unit (CPU); a graphical processing unit (GPU); or other processing units.

[0073] Embodiment 9 is the method of any of embodiments 1-8, wherein the one or more pages includes a page adjacent to a page containing the virtual address.

[0074] Embodiment 10 is the method of any of embodiments 1-9, wherein the programmable prefetching policy includes one of; (a) whether to perform prefetching; (b) a number of pages to prefetch; (c) a direction of pages relative to the virtual address to prefetch; (d) an interval to jump between pages to prefetch; or (e) any combination of (a), (b), (c), and (d).

[0075] Embodiment 11 is the method of any of embodiments 1-10, wherein injecting the prefetch request into the stream of memory access requests, further comprises: for a given cycle, determining that the MMU did not receive a valid memory access request and that the prefetch request in the prefetch buffer is valid; and injecting the prefetch request for the given cycle.

[0076] Embodiment 12 is the method of any of embodiments 1-11, wherein injecting the prefetch request into the stream of memory access requests, further comprises: for a given cycle determining that the MMU received a valid memory translation request and that the prefetch buffer includes a valid prefetch request; and injecting either the valid memory translation request or the prefetch request for the given cycle based on a defined arbitration scheme.

[0077] Embodiment 13 is the method of embodiment 12, wherein the defined arbitration scheme uses a weighted round robin.

[0078] Embodiment 14 is the method of embodiment 13, wherein weights for the weighted round robin are determined based on the programmable prefetching policy.

[0079] Embodiment 15 is a memory management unit (MMU) comprising: an address translation buffer unit) including a translation lookaside buffer (TLB); an address translation control unit configured to perform translation table walks; wherein the MMU is configured to: receive a memory access request in a stream of memory access requests, the memory access request including a virtual address and at least one non-address tag; determine that a mapping for the virtual address is not cached in the TLB; generating a prefetch request based on thedetermination that the mapping for the virtual address is not cached in the TLB, wherein generating the prefetch request includes: retrieve a programmable prefetching policy from a look-up table based on the at least one non-address tag; determine one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request; and store the prefetch request in a prefetch buffer; and inject the prefetch request from the prefetch buffer into the stream of memory access requests to preload one or more address translations for the one or more pages in the TLB.

[0080] Embodiment 16 is the MMU of embodiment 15 wherein the MMU is a system memory management unit (SMMU).

[0081] Embodiment 17 is the MMU of embodiment 15 or 16, wherein the MMU includes a plurality of address translation buffer units.

[0082] Embodiment 18 is the MMU of embodiment 17, wherein the one or more pages included in the prefetch request are determined on a per address translation buffer unit basis for the plurality of address translation buffer units.

[0083] Embodiment 19 is the MMU of any of embodiments 15-18, wherein the MMU disables a clock for the address translation control unit for at least a portion of time where there are no TLB misses.

[0084] Embodiment 20 is a computing system comprising: a processing unit; a memory management unit (MMU) comprising a translation lookaside buffer (TLB) and configured to perform address translations for the processing unit including to: receive a memory access request in a stream of memory access requests from the processing unit, the memory access request including a virtual address and at least one non-address tag; determine that a mapping for the virtual address is not cached in the TLB; generating a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB, wherein generating the prefetch request includes: retrieve a programmable prefetching policy from a look-up table based on the at least one non-address tag; determine one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request; and store the prefetch request in a prefetch buffer; and inject the prefetch request from the prefetch buffer into the stream of memory access requests to preload one or more address translations for the one or more pages in the TLB.

[0085] Embodiment 21 is the computing system of embodiment 20, wherein the computing system is part of a mobile computing device and a plurality of prefetch requests in the prefetch buffer are updated to remove overlapping regions to preserve power of the mobile computing device.

[0086] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment.Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0087] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0088] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: receiving, at a memory management unit (MMU), a memory access request in a stream of memory access requests, the memory access request including a virtual address and at least one non-address tag; determining that a mapping for the virtual address is not cached in a translation lookaside buffer (TLB) of the MMU; generating a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB, wherein generating the prefetch request includes: retrieving a programmable prefetching policy from a look-up table based on the at least one non-address tag; determining one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request; and storing the prefetch request in a prefetch buffer; and injecting the prefetch request from the prefetch buffer into the stream of memory access requests to preload one or more address translations for the one or more pages in the TLB.

2. The method of claim 1, wherein determining the one or more pages to prefetch includes using the programmable prefetching policy to identify pages that are likely to be accessed by upcoming memory access requests.

3. The method of claim 1, wherein storing the prefetch request in the prefetch buffer further comprises: determining whether the prefetch request includes an overlapping region with a second prefetch request in the prefetch buffer; and updating the prefetch request to remove the overlapping region.

4. The method of claim 3, wherein determining whether the prefetch request includes the overlapping region with the second prefetch request in the prefetch buffer further comprises:determining that the non-address tag of the memory access request is a match of a second non-address tag of the second prefetch request; and comparing a first region for the memory access request with a second region for the second prefetch request to determine the overlapping region.

5. The method of claim 4, wherein the MMU uses a power efficient sieve to compare the first region with the second region.

6. The method of claim 1, wherein the at least one non-address tag includes an identifier for an application requesting the memory access request.

7. The method of claim 1, wherein the at least one non-address tag includes an identifier for a source of the memory access request.

8. The method of claim 7, wherein the source one of a central processing unit (CPU); or a graphical processing unit (GPU); a tensor processing unit (TPU); an Al accelerator processing unit; or an application specific processing unit.

9. The method of claim 1, wherein the one or more pages includes a page adjacent to a page containing the virtual address.

10. The method of claim 1, wherein the programmable prefetching policy includes one of:(a) whether to perform prefetching;(b) a number of pages to prefetch;(c) a direction of pages relative to the virtual address to prefetch;(d) an interval to jump between pages to prefetch; or(e) any combination of (a), (b), (c), and (d).11 . The method of claim 1, wherein injecting the prefetch request into the stream of memory access requests, further comprises: for a given cycle, determining that the MMU did not receive a valid memory access request and that the prefetch request in the prefetch buffer is valid; and injecting the prefetch request for the given cycle.

12. The method of claim 1, wherein injecting the prefetch request into the stream of memory access requests, further comprises: for a given cycle determining that the MMU received a valid memory translation request and that the prefetch buffer includes a valid prefetch request; and injecting either the valid memory translation request or the prefetch request for the given cycle based on a defined arbitration scheme.

13. The method of claim 12, wherein the defined arbitration scheme uses a weighted round robin.

14. The method of claim 13, wherein weights for the weighted round robin are determined based on the programmable prefetching policy.

15. A memory management unit (MMU) comprising: an address translation buffer unit including a translation lookaside buffer (TLB); and an address translation control unit configured to perform translation table walks; wherein the MMU is configured to: receive a memory access request in a stream of memory access requests, the memory access request including a virtual address and at least one non-address tag; determine, by the address translation buffer unit, that a mapping for the virtual address is not cached in the TLB; generate a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB, wherein to generate the prefetch request includes to: retrieve a programmable prefetching policy from a look-up table based on the at least one non-address tag;determine one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request; and store the prefetch request in a prefetch buffer; inject the prefetch request from the prefetch buffer into the stream of memory access requests; and perform, by the address translation control unit, a translation table walk for the prefetch request to preload one or more address translations for the one or more pages in the TLB.

16. The MMU of claim 15 wherein the MMU is a system memory management unit (SMMU).

17. The MMU of claim 15, wherein the MMU includes a plurality of address translation buffer units.

18. The MMU of claim 17, wherein the one or more pages included in the prefetch request are determined on a per address translation buffer unit basis for the plurality of address translation buffer units.

19. The MMU of claim 15, wherein the MMU disables a clock for the address translation control unit for at least a portion of time where there are no TLB misses.

20. A computing system comprising: a processing unit; and a memory management unit (MMU) comprising a translation lookaside buffer (TLB) and configured to perform address translations for the processing unit including to: receive a memory access request in a stream of memory access requests from the processing unit, the memory access request including a virtual address and at least one nonaddress tag; determine that a mapping for the virtual address is not cached in the TLB;generate a prefetch request based on the determination that the mapping for the virtual address is not cached in the TLB, wherein to generate the prefetch request includes to: retrieve a programmable prefetching policy from a look-up table based on the at least one non-address tag; determine one or more pages to prefetch with the prefetch request based on the programmable prefetching policy and the virtual address of the memory access request; and store the prefetch request in a prefetch buffer; and inject the prefetch request from the prefetch buffer into the stream of memory access requests to preload one or more address translations for the one or more pages in the TLB.

21. The computing system of claim 20, wherein the computing system is part of a mobile computing device and a plurality of prefetch requests in the prefetch buffer are updated to remove overlapping regions to preserve power of the mobile computing device.

Images