Address Translation Service Buffer

Abstract

An Address Translation Buffer, or ATB, is provided to emulate or implement the Peripheral Component Interface Express (PCIe) Address Translation Services (ATS) protocol within a PCIe-compliant device. The ATB operates in place of (or in addition to) an Address Translation Cache (ATC), but is implemented in firmware or hardware without requiring the robust set of resources associated with a persistent hardware cache (e.g., circuitry for cache control and lookup). A component of the device (e.g., a DMA engine), for example via a host I / O memory management unit, requests translation of an untranslated address, and the response (including the translated address) is stored in the ATB for use in a single DMA operation (which may involve multiple transactions across the PCIe bus).

Description

Background Art

[0001] An electronic device operating within a PCIe (Peripheral Component Interface Express) fabric may implement the PCIe ATS (Address Translation Services) protocol to facilitate the translation between untranslated (virtual) addresses and translated (physical) addresses. These devices typically include a hardware ATC (Address Translation Cache) that caches translations for reuse.

[0002] However, a hardware cache requires dedicated memory and corresponding circuitry to control the cache, perform lookups, and / or support other operations. For example, when a cache (or a processor or controller for the cache) receives an invalidation request, the cache needs to search for matching entries and perform appropriate actions. Thus, implementing and supporting a hardware cache requires resources that could be used for other purposes. Further, depending on the environment, such as when a significant percentage of cached translations are only used once before being invalidated or evicted from the cache, the ATC can be inefficient.

Brief Description of the Drawings

[0003] [Figure 1] A block diagram of an electronic device implementing an address translation buffer, according to some embodiments. [Figure 2] A flowchart showing a method of using an address translation buffer, according to some embodiments. [Figure 3A] A flowchart showing an alternative process for handling an invalidation request for an address translation buffer within an electronic device, according to some embodiments. [Figure 3B] A flowchart showing an alternative process for handling an invalidation request for an address translation buffer within an electronic device, according to some embodiments. [Modes for carrying out the invention]

[0004] The following descriptions are provided to enable any person skilled in the art to implement and use the disclosed embodiments, and are provided in the context of one or more specific uses and their requirements. Various modifications to the disclosed embodiments will be readily apparent to a person skilled in the art, and the general principles described herein may be applied to other embodiments and uses without departing from the scope of what is disclosed. Accordingly, the present invention is not intended to be limited to the embodiments shown, but rather the broadest scope consistent with this disclosure should be recognized.

[0005] (Address translation buffer) In some embodiments, an Address Translation Buffer (ATB) is provided within a device operating as part of a PCIe (Peripheral Component Interface Express) fabric to facilitate address translation according to the PCIe ATS (Address Translation Services) protocol. In these embodiments, the ATB is configured to have lower cost and less silicon area while achieving equivalent or better performance compared to an ATC, without requiring dedicated hardware resources and the complexity associated with a conventional Address Translation Cache (ATC). The ATB may be implemented in firmware. In this case, the ATB may require no dedicated hardware resources at all, or may be implemented in hardware without the resource footprint required by an ATC.

[0006] An ATB can exemplify a ring buffer implementing a first-in, first-out (FIFO) queue, but other forms or formats may be used. For example, one alternative implementation could be a content-associative memory or queue. This alternative implementation is particularly suitable for environments where address translations are received in no particular order (with respect to the order of each translation request) and cannot be sorted, as entries (i.e., address translations) are searchable using untranslated addresses.

[0007] The essence of a device using ATB is not limited to the fact that it is of a type that requires the translation of untranslated addresses (UTAs) to translated addresses (TAs). For example, a device may function to move data between the device and a host computer system, between storage locations within the host system, between virtual machines running on the host system, between virtual machines and hypervisors, between different processing threads, between threads and other entities, etc. Therefore, in some exemplary embodiments, a device may be, or include, a data controller, a network interface, a storage (e.g., disk, SSD) controller, a compression engine, a graphics controller, an encryption controller, etc.

[0008] Figure 1 is a block diagram of an electronic device implementing an address translation buffer according to one or more embodiments described herein.

[0009] In these embodiments, device 100 operates within or with a host system that makes various resources accessible to the device via the PCIe bus 150 and / or other paths. Host resources may include a central processing unit (CPU), an input / output memory management unit (IOMMU), primary and / or secondary storage devices, a communications controller (e.g., network, USB), a graphics controller, and the like.

[0010] Device 100 features one or more processors or microcontrollers 102 (e.g., processors 102A to 102N), a bus 104, and a DMA (Direct Memory Access) engine 110. Other device components and interfaces are omitted for clarity. The DMA engine 110 includes local registers 112 and an address translation buffer (ATB) 114. The DMA engine operates according to instructions executed by one or more of the processors 102.

[0011] While device 100 is operating, processor 102 instructs DMA engine 110 to translate one or more sets of untranslated (e.g., virtual) addresses in order to enable DMA operation (or for some other reason). Accordingly, DMA engine 110 issues one or more corresponding translation requests to the host IOMMU via PCIe bus 150, receives one or more responses containing the corresponding translated (e.g., physical) addresses, and stores the responses in ATB 114. The address translation buffer may be configured to store a different number (e.g., 64, 128, 256) of address translation responses in different implementations. Buffered responses may be identified by an identifier (e.g., job ID), untranslated addresses, translated addresses, and / or other information.

[0012] Eventually, the processor may receive or retrieve the buffered response and use it to initiate the corresponding I / O (e.g., reading or writing to memory) or for some other purpose. Once the processor receives a response from the ATB, the response may be temporarily stored in the processor cache or other temporary storage for the processor (e.g., static random access memory, i.e., SRAM) before being used to initiate or execute the associated memory access.

[0013] In some embodiments, the functions of the DMA engine 110 may be divided among several separate components. For example, one component (e.g., a translation component) may be responsible for issuing address translation requests, storing the responses in the ATB, and then distributing each response to the processor and / or a second component. The second component (e.g., a DMA component) may be responsible for using the responses to perform data transfer or copying. In these embodiments, the ATB 114 may be considered to span both components.

[0014] The local register 112 temporarily stores signals and messages sent to and from the host system via the PCIe bus 150 to support the emulation or implementation of ATS. Although shown separately from the address translation buffer 114 in Figure 1, in some embodiments the local register 112 may encompass the ATB 114.

[0015] The ATB114 can span multiple clock domains. For example, the processor 102 operates within a first clock domain, while part or all of the DMA engine 110 (e.g., the interface between the DMA engine and the PCIe bus 150) operates within a second domain.

[0016] It should be noted that in some implementations or situations, responses to address translation requests may be used or consumed immediately in some way, rather than being buffered. For example, instead of storing newly received responses in the ATB, the responses may be delivered directly to the device processor 102. The processor may use the responses without storing them in the ATB (e.g., to initiate I / O operations via the DMA engine 110), relay the responses to the ATB for storage after inspection, or temporarily store the responses before further processing (e.g., in the processor cache or static RAM).

[0017] In one or more embodiments, the device 100 may include an address translation cache (ATC) in addition to the ATB 114, which is not shown in Figure 1. In these embodiments, address translation responses that are likely to be reused may be stored in the ATC, while responses that are unlikely to be reused may be stored in the ATB.

[0018] Figure 2 is a flowchart illustrating the process for using an address translation buffer according to one or more embodiments described herein.

[0019] The illustrated process begins with programming the DMA engine of the electronic device by the host system processor and / or a processor or controller present on the device (step 200). The programming steps may include loading the device driver, instantiating and / or configuring the address translation buffer, initializing the local registers of the DMA engine, and so on.

[0020] Next, the desired DMA operation is identified (step 202). This is sometimes done by the device processor, including the untranslated addresses. A single DMA operation may ultimately require multiple moves of data across the device's PCIe bus. Specifically, the target DMA operation may involve transferring more data than can be handled by a single bus transaction. Therefore, while the device processor may perceive the transfer as a single operation, the DMA engine may have to perform several transactions using a single translated address (appropriately incremented during the execution of individual bus transactions).

[0021] The DMA engine simultaneously issues one or more corresponding translation requests to the host system (e.g., the host IOMMU) (step 204). Note that a single DMA operation may require multiple address translations. For example, multipage data transfer may require separate translations for each page.

[0022] After that, a response including the translated address related to the untranslated address is received and stored in the ATB for each translation request (step 206). In addition to the translated address, the response may include related metadata stored in the ATB together with the translated address for the purpose of matching the translation response to the corresponding translation request and / or for other purposes. Such metadata may include, by way of example, an identifier or tag, the untranslated address, a timestamp, etc.

[0023] At some point in the future, using one or more translated addresses read from the ATB, a DMA operation corresponding to the response is performed (step 208). For example, the response may be transferred to a processor that instructs the DMA engine to perform a read / write of memory or some other operation using the translated address and / or other information within the response. After completion of the DMA operation, the corresponding address translation is purged from the ATB (step 210), and the illustrated method ends.

[0024] Figures 3A - 3B are flow diagrams showing an alternative process for handling an invalidation request within an electronic device implementing an address translation buffer according to one or more embodiments described herein. The process shown in Figure 3A generally requires less processing overhead compared to the process shown in Figure 3B, but may take longer depending on the operating environment. For example, if there are no buffered translation requests that match the invalidation request, the process shown in Figure 3B may allow for a quicker response to the invalidation request.

[0025] The process of Figure 3A begins with receiving an invalidation request in the DMA engine within the electronic device. This request specifies the memory address to be invalidated (step 300). In response, a snapshot of the contents of the ATB of the DMA engine is generated (step 302). The snapshot stores all the ATB entries / responses stored prior to the invalidation request.

[0026] Next, the device deletes the address translation response at the head of the buffer (step 304), processes the response to perform one or more memory accesses and / or other related actions (step 306), and determines whether all responses for which a snapshot was generated have been flushed from the ATB (step 308), thereby flushing the contents of the buffer for which the snapshot was generated. Steps 304-308 may be repeated as necessary.

[0027] When all address translation responses buffered upon receipt of an invalidation request have been processed and the related DMA operations involving address translation are complete, the device responds to the invalidation request (step 310). Note that none, some, or all of the responses for which a snapshot was generated may include the address specified in the invalidation request. After step 310, the process of FIG. 3A is complete.

[0028] The process of FIG. 3B begins with receiving an invalidation request in the DMA engine within the electronic device (step 350). This request is examined to identify the memory addresses to be invalidated (step 352).

[0029] In these embodiments, the address translation responses stored in the ATB may include both the untranslated and translated addresses included in the corresponding address translation requests dispatched to the host IOMMU (or other host component), and / or other data or metadata for matching the address translation response to the corresponding translation request.

[0030] After identifying the target addresses, the ATB is scanned to identify all entries (address translation responses) that match the invalidation request (step 354). A snapshot of the identified entries (if any) is generated (step 356).

[0031] The process removes the address translation response at the beginning of the ATB and processes it to perform memory access and / or other related actions (step 360) until all entries that generated a snapshot have been processed (step 358). The process then returns to step 358. Once all entries that generated a snapshot have been processed, the device responds to the invalidation request (step 362). Note that if there are no ATB entries that match the invalidation request, step 360 is never performed. In this case, no snapshot of the response is generated in step 356. This means that step 362 follows immediately after step 356 or 358.

[0032] In some embodiments, at least one electronic device (e.g., electronic device 100) uses code and / or data stored in a non-temporary computer-readable storage medium to perform some or all of the operations described herein. More specifically, at least one electronic device reads code and / or data from the computer-readable storage medium and executes the code and / or uses the data to perform the operations described herein. The computer-readable storage medium may be any device, medium, or combination thereof that stores the code and / or data used by the electronic device. For example, the computer-readable storage medium may include, but is not limited to, flash memory, random access memory (e.g., eDRAM, RAM, SRAM, DRAM, DDR4 SDRAM, etc.), non-volatile RAM (e.g., phase-change memory, ferroelectric random access memory, spin-transfer random access memory, magnetoresistive random access memory, etc.), read-only memory (ROM), and / or magnetic or optical storage medium (e.g., disk drive, magnetic tape, CD, DVD, etc.).

[0033] In some embodiments, one or more hardware modules perform the operations described herein. For example, a hardware module may include, but is not limited to, one or more central processing units (CPUs) / CPU cores, graphics processing units (GPUs) / GPU cores, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), compressors or encoders, compute units, embedded processors, accelerated processing units (APUs), controllers, and / or other functional blocks. When a circuit configuration within such a hardware module (e.g., integrated circuit elements, discrete circuit elements, etc.) is activated, the circuit configuration performs some or all of the operations. In some embodiments, a hardware module includes general-purpose circuits such as execution pipelines, compute or processing units that perform operations when instructions (e.g., program code, firmware, etc.) are executed. In some embodiments, a hardware module includes application-specific or dedicated circuit configurations that perform operations, and may include circuit configurations that perform some or all of the operations "in hardware" without executing instructions.

[0034] In some embodiments, data structures representing some or all of the functional blocks and circuit elements described herein (e.g., electronic device 100, processor 102, DMA engine 110) are stored in a non-temporary computer-readable storage medium containing a database or other data structures that can be read by the electronic device and used directly or indirectly for fabricating hardware including the functional blocks and circuit elements. For example, the data structures may be behavioral-level descriptions or register-transfer-level (RTL) descriptions of hardware functions in a high-level design language (HDL) such as Verilog or VHDL. The descriptions may be read by a synthesis tool that can synthesize the descriptions to generate a netlist containing a list of transistors / circuit elements from a synthesis library representing the functions of the hardware including the functional blocks and circuit elements. The netlist may then be arranged and routed to generate a dataset describing the geometric shapes to be applied to a mask. The mask may then be used in various semiconductor fabrication steps to manufacture semiconductor circuits or circuits (e.g., integrated circuits) corresponding to the functional blocks and circuit elements. Alternatively, the database on a computer-accessible storage medium may, as desired, be a netlist (with or without a synthetic library), a dataset, or Graphic Data System (GDS) II data.

[0035] In this description, variables or unspecified values ​​(i.e., general descriptions of values ​​that do not include specific examples of values) are represented by letters such as N, M, and X. Although similar letters may be used in different places in this description as used herein, the variables and unspecified values ​​in each case are not necessarily the same; that is, there may be intended variables and values ​​for some or all of the general variables and unspecified values. In other words, specific examples of values ​​for N, as well as any other letters used to represent variables and unspecified values ​​in this description, are not necessarily related to one another.

[0036] Where used herein, the expressions “other” or “etc.” are intended to present an equivalent of one and / or example, i.e., at least one of the elements in the list to which “other” is associated. For example, the statement “an electronic device performs a first operation, a second operation, etc.” means that the electronic device performs at least one of the first operation, a second operation and other operations. In addition, the elements in the list associated with one, etc. are merely examples from a set of examples, and at least some of the embodiments may not appear in some embodiments.

[0037] The embodiments described above are presented for illustrative and explanatory purposes only. They are not intended to be exhaustive or to limit this disclosure to the disclosed forms. Many modifications and variations will therefore be apparent to those skilled in the art. The scope of the embodiments is defined not by the above disclosure but by the appended claims.

Claims

1. It is an electronic device, Address translation cache, Address translation buffer, Equipped with a Direct Memory Access (DMA) engine, The aforementioned DMA engine is To issue a translation request regarding untranslated addresses to the host computer system, Receiving an address translation that includes the translated address corresponding to the aforementioned untranslated address, The address translation is stored in the address translation buffer, It is configured to do, Electronic devices.

2. The aforementioned DMA engine is Deleting the address translation from the address translation buffer, The DMA operation is initiated using the aforementioned translated address, It is configured to do, The electronic device according to claim 1.

3. The aforementioned DMA engine is A translation component configured to issue the aforementioned translation request and receive the aforementioned address translation, A DMA component configured to initiate DMA operation, The electronic device according to claim 1.

4. The DMA engine includes the address translation buffer, The electronic device according to claim 1.

5. The address translation buffer is implemented as a first-in, first-out queue. The electronic device according to claim 1.

6. The aforementioned address translation buffer is implemented as content-associative memory. The electronic device according to claim 1.

7. The aforementioned address translation is purged from the address translation buffer when in use. The electronic device according to claim 1.

8. The aforementioned DMA engine is Receiving a deactivation request from the aforementioned host computer system, To generate a snapshot of the address translation stored in the address translation buffer, The address translation for which a snapshot was generated is flushed from the address translation buffer, It is configured to do, The electronic device according to claim 1.

9. The aforementioned DMA engine is Receiving a deactivation request from the host computer system, wherein the deactivation request identifies one or more target addresses, Identifying an address translation stored in the address translation buffer that matches one or more target addresses, The address translations are flushed from the address translation buffer until all identified address translations are removed from the address translation buffer. It is configured to do, The electronic device according to claim 1.

10. A method for buffering address translation in an electronic device comprising an address translation buffer and an address translation cache, Receiving an untranslated address, Dispatching a request to the host computer system to translate the aforementioned untranslated address, Receiving an address translation from the host computer system that includes a translated address corresponding to the untranslated address, The method includes storing the address translation in the address translation buffer, method.

11. Deleting the address translation from the address translation buffer, Distributing the address translation to the processor present in the electronic device, This includes initiating a direct memory access operation using the translated address, The method of claim 10.

12. Receiving a deactivation request from the aforementioned host computer system, To generate a snapshot of the address translation stored in the address translation buffer, This includes flushing the address translation for which a snapshot was generated from the address translation buffer, The method of claim 10.

13. Receiving a deactivation request from the host computer system, wherein the deactivation request identifies one or more target addresses, Identifying an address translation stored in the address translation buffer that matches one or more target addresses, This includes flushing the address translations from the address translation buffer until all identified address translations are removed from the address translation buffer, The method of claim 10.

14. A computer-readable storage medium for storing instructions, When the instruction is executed by the processor, it causes the processor to perform a method of buffering address translations within an electronic device comprising an address translation buffer and an address translation cache. The aforementioned method, Receiving an untranslated address, Dispatching a request to the host computer system to translate the aforementioned untranslated address, Receiving an address translation from the host computer system that includes a translated address corresponding to the untranslated address, The method includes storing the address translation in the address translation buffer, Computer-readable storage medium.

15. The aforementioned method, Deleting the address translation from the address translation buffer, Distributing the address translation to the processor present in the electronic device, This includes initiating a direct memory access operation using the translated address, A computer-readable storage medium according to claim 14.

16. The address translation buffer is implemented as a first-in, first-out queue. A computer-readable storage medium according to claim 14.

17. The aforementioned address translation is purged from the address translation buffer when in use. A computer-readable storage medium according to claim 14.

18. The aforementioned method, Receiving a deactivation request from the aforementioned host computer system, To generate a snapshot of the address translation stored in the address translation buffer, This includes flushing the address translation for which a snapshot was generated from the address translation buffer, A computer-readable storage medium according to claim 14.

19. The aforementioned method, Receiving a deactivation request from the host computer system, wherein the deactivation request identifies one or more target addresses, Identifying an address translation stored in the address translation buffer that matches one or more target addresses, This includes flushing the address translations from the address translation buffer until all identified address translations are removed from the address translation buffer, A computer-readable storage medium according to claim 14.

20. The address translation buffer has a reduced resource footprint compared to the address translation cache. The electronic device according to claim 1.

Images