Page table cache optimization method and page table update method

Abstract

A page table cache optimization method, a page table update method and a device are disclosed. Before creating a module on a physical machine, the module's attribute information and / or currently available page table cache resources are obtained. The attribute information is information related to the module's demand for the page table cache. A page table cache of a first page size is configured for the module, and the first page size is set based on the attribute information and / or the page table cache resources. In this way, the page table cache of the modules created on the physical machine can be evenly distributed while fully considering the module characteristics and the system's available resources.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to a page table cache optimization method and a page table update method. Background Art

[0002] The traditional MMU (Memory Management Unit) converts the virtual address accessed by the CPU into the actual physical address. The IOMMU (Input / Output Memory Management Unit) converts the virtual address of the virtual machine accessed by the device into the physical address.

[0003] In the virtualization of pass-through devices, the mapping between device virtual addresses (IOVA) and real physical addresses (HPA) can be achieved through IOMMU. This mapping relationship is stored in the system's IOMMU page table. Like the traditional MMU page table, the IOMMU page table is also composed of multiple levels of page tables. The mapping relationship from IOVA to HPA can be obtained by checking the IOMMU page table level by level.

[0004] In order to improve the table lookup efficiency, similar to MMUTLB, the IOMMU side also introduces the IOMMU TLB to reduce the number of table lookups, thereby improving the system's IO performance.

[0005] Whether it is MMU TLB or IOMMU TLB, due to the limited TLB cache space, TLB caches only some of the page table entries in the page table. If the address to be queried is not in the page table entry cached by TLB, it will cause a TLB miss, that is, TLB query failure. The cost of TLB query failure is relatively high.

[0006] Therefore, a TLB optimization solution that can reduce TLB misses is needed. Summary of the invention

[0007] A technical problem to be solved by the present disclosure is to provide a TLB optimization solution capable of reducing TLB misses.

[0008] According to a first aspect of the present disclosure, a page table cache optimization method is provided, comprising: before creating a module on a physical machine, obtaining attribute information of the module and / or currently available page table cache resources, the attribute information being information related to the degree of demand for page table cache by the module; configuring a page table of a first page size for the module, the first page size being set based on the attribute information and / or the page table cache resources.

[0009] Optionally, the attribute information includes priority and / or memory specification.

[0010] Optionally, the size of the first page size is positively correlated with the priority and / or memory specification of the module.

[0011] Optionally, the virtual address space of the module is divided into multiple pages, and the page size of the page is the same as the first page size. The method also includes: generating a bitmap corresponding to the module, each bit in the bitmap corresponds to a page in the multiple pages; recording the page corresponding to the virtual address accessed by the address access operation initiated for the virtual address space of the module within a predetermined time range during the operation of the module; setting the bit in the bitmap corresponding to the recorded page; and evaluating the performance of the page table cache based on the setting of the bit in the bitmap.

[0012] Optionally, the method further includes: adjusting the first page size to a second page size based on the evaluation result.

[0013] Optionally, the method further includes: redetermining the number of page table levels of the page table according to the second page size; and regenerating the page table corresponding to the module based on the redetermined number of page table levels.

[0014] Optionally, the module is a virtual machine, the address access operation is initiated by a device directly connected to the virtual machine, the virtual machine runs in a virtualization program, and the method also includes: creating a new virtualization program; stopping the operation of the virtual machine and the operation of the device directly connected to the virtual machine; redetermining the page table level of the page table according to the second page size; generating a page table corresponding to the module in the new virtualization program according to the redetermined page table level; remapping the memory of the old virtualization program to the new virtualization program; and resuming the operation of the virtual machine and the operation of the device directly connected to the virtual machine.

[0015] According to a second aspect of the present disclosure, a page table update method is provided, comprising: during the operation of a module created on a physical machine, evaluating the performance of a page table cache of the module; and determining whether it is necessary to adjust the page size of a page in the page table of the module based on the evaluation result.

[0016] Optionally, the step of evaluating the performance of the page table cache of the module includes: generating a bitmap corresponding to the module, each bit in the bitmap corresponding to a page in the page table; recording, during the operation of the module, pages corresponding to virtual memory addresses accessed by direct memory access operations initiated for the virtual address space of the module within a predetermined time range; setting bits in the bitmap corresponding to the recorded pages; and evaluating the performance of the page table cache based on the settings of the bits in the bitmap.

[0017] Optionally, the method also includes: if it is determined that the page size of a page in the page table of the module needs to be adjusted, adjusting the page size from the first page size to the second page size; redetermining the page table level of the page table according to the second page size; and regenerating the page table corresponding to the module based on the redetermined page table level.

[0018] According to a third aspect of the present disclosure, a page table cache optimization device is provided, including: an acquisition module, used to acquire attribute information of a module and / or currently available page table cache resources before creating the module on a physical machine, the attribute information being information related to the degree of demand for page table cache by the module; and a configuration module, used to configure a page table of a first page size for the module, the first page size being set based on the attribute information and / or the page table cache resources.

[0019] According to a fourth aspect of the present disclosure, a page table updating device is provided, comprising: an evaluation module for evaluating the performance of a page table cache of a module during the operation of the module created on a physical machine; and a determination module for determining whether the page size of a page in the page table of the module needs to be adjusted based on the evaluation result.

[0020] According to a fifth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in the first aspect or the second aspect above.

[0021] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising an executable code, which, when executed by a processor of an electronic device, enables the processor to execute the method described in the first aspect or the second aspect above.

[0022] According to a seventh aspect of the present disclosure, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first or second aspect above.

[0023] Therefore, the present invention obtains the module's attribute information related to the degree of demand for page table cache and / or the currently available page table cache resources, and sets the page size of the page table configured for the module based on the attribute information and / or the page table cache resources, so that balanced allocation of TLBs of modules created on the physical machine can be achieved while fully considering the characteristics of the modules and the available resources of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0025] Figure 1 A schematic flow chart of a page table cache optimization method according to an embodiment of the present disclosure is shown.

[0026] Figure 2A schematic diagram of the tracking principle of the IOTLB access model according to an embodiment of the present disclosure is shown.

[0027] Figure 3 A schematic diagram of using a bitmap to record accessed memory ranges according to an embodiment of the present disclosure is shown.

[0028] Figure 4 A schematic structural diagram of a page table cache optimization device according to an embodiment of the present disclosure is shown.

[0029] Figure 5 A schematic structural diagram of a page table updating device according to an embodiment of the present disclosure is shown.

[0030] Figure 6 A schematic diagram of the structure of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] On a physical machine, you can create various types of modules, including but not limited to processes and virtual machines. For security and versatility, modules use virtual addresses to access memory, which need to be converted to real physical addresses to obtain data from memory. The mapping relationship between virtual addresses and real physical addresses is stored in the page table set for the module.

[0033] Specifically, each module is assigned an available physical address space, and the virtual address spaces used by different modules are isolated from each other. The virtual address space and the physical address space are divided into pages and page frames according to fixed sizes (such as 4KB). The value obtained by dividing the virtual address space by the page size (that is, the page size, such as 4KB) is the number of pages. The virtual address consists of two parts, namely the virtual page number (VPN) and the offset within the page (Offset). Correspondingly, the physical address also consists of two parts, namely the page frame number (also called the physical page number, PhysicalFrame Number, PFN) and the offset within the page (Offset). Therefore, mapping the virtual address to the physical address is the process of mapping the page number of a virtual address to the page number of a physical address, and the page table records the mapping relationship between the virtual page number and the physical page number. The page table includes multiple page table entries, each of which is used to represent the corresponding relationship between a virtual page number and a physical page number.

[0034] In order to reduce the memory space occupied by the page table itself, the page table generally adopts a multi-level page table. The price paid for this is that the number of memory accesses during the address translation process increases, thereby increasing the translation time. The multi-level page table is a mature technology in this field and will not be described here.

[0035] In order to reduce the translation time, some page table entries of the page table (such as the page table entries involved in the most recent address access operation) are stored in a high-speed cache called the Translation Lookaside Buffer (TLB). Simply put, TLB is a cache of the page table, which stores the page table entries that are most likely to be accessed at the moment. Its content is a copy of some page table entries in the page table, so TLB can also be called a page table cache, or a page table mapping cache. For a page table entry that is not in the TLB cache, attempting to access the address contained in the page corresponding to the page table entry will result in a TLB query failure, that is, a TLB miss. The cost of a TLB query failure is relatively high.

[0036] In order to allow the TLB to store more page address mapping relationships, the general practice is to increase the page size used when dividing pages. The page size usually used when dividing pages is 4KB. In order to allow the TLB to store more page address mapping relationships, a large page size (such as 2MB, 1GB) can be used to divide the address space into multiple continuous pages. At this time, each page table entry in the obtained page table can contain more page address mapping relationships, and the corresponding TLB can also cache more page address mapping relationships, thereby reducing the number of TLB misses. Compared with the standard page size (such as 4KB), the number of pages divided using a large page size is smaller, so the number of page table levels can be relatively reduced, so that when a TLB miss occurs, the number of page table levels that need to be queried after a TLB miss (i.e., the number of queries) can also be reduced, thereby improving the query efficiency after a miss.

[0037] For example, when using 2MB large pages compared to traditional 4KB page tables, if continuous DMA (Direct Memory Access) requests spanning 10 4KB physical address ranges are initiated, 10 TLB misses will be generated. If 2MB large pages are used, although the 10 accesses span multiple 4KB physical address ranges, they are very likely to belong to the same 2MB physical page, and therefore only one TLB miss will be triggered. Moreover, for a TLB miss in the 4KB page table, it is usually necessary to query the 3-level page table of level 1 to level 3. For 2MB or 1GB large pages, due to the small number of page table levels, a TLB miss only needs to query the level 2 or level 1 page table to complete.

[0038] However, when configuring page tables for modules, the performance of TLBs with large pages (such as 1GB) is not necessarily better than that of TLBs with small pages (such as 4KB, 2MB). For example, when a module has frequent cross-page accesses (such as memory accesses that span 10GB), even if a page table with a 1GB page size is configured for the module, the number of 1GB large page table entries that the TLB can cache is relatively small, making the TLB miss probability of the 1GB page size TLB cache still high. In this case, the performance of the TLB cache with a 2M large page will be better. For another example, when creating a large number of modules on a physical machine, a TLB needs to be configured for each module, and the TLB resources on the physical machine are limited. Therefore, if a 1GB large page page table is configured for each program or virtual machine, when multiple programs or virtual machines perform address access operations (such as DMA operations) at the same time, the TLB will be frequently swapped in and out, resulting in a serious performance degradation.

[0039] In view of this, the present disclosure proposes that a page table with an appropriate page size can be set for the module based on the module's attribute information and / or the currently available TLB resources (i.e., page table cache resources) so as to achieve balanced allocation of the TLBs of the modules created on the physical machine while fully considering the module characteristics and the system's available resources.

[0040] Furthermore, the present disclosure also proposes a method for real-time monitoring (ie, real-time evaluation) of the TLB performance of a module during operation, so as to dynamically update the page table of the module according to the monitoring result (evaluation result), thereby achieving dynamic optimization of the TLB.

[0041] The details of the present disclosure are further described below in conjunction with the accompanying drawings.

[0042] Figure 1 A schematic flow chart of a page table cache optimization method according to an embodiment of the present disclosure is shown.

[0043] See also Figure 1 In step S110, before creating the module on the physical machine, the attribute information of the module and / or the currently available page table cache resources are obtained.

[0044] The present disclosure can be used to optimize both the traditional MMUTLB and the IOMMUTLB. When used to optimize the traditional MMUTLB, the module can refer to a process, and when used to optimize the IOMMUTLB, the module can refer to a virtual machine.

[0045] The attribute information of a module refers to information related to the module's need for page table cache, that is, information that can characterize the module's need for page table cache. For example, it can be information that can reflect the module's memory specifications and / or priority. That is, the attribute information can include the module's priority and / or memory specifications.

[0046] One or more modules can be created on a physical machine. If multiple modules need a resource at the same time, the priority of allocating resources to a module will be determined based on the importance of the module. The priority can reflect the priority of the module in obtaining resources. The priority of the module can be determined based on the importance of the module. The higher the importance of the module, the greater the priority.

[0047] As an example, the priority of a module may be determined according to the function and / or type of the module. For example, the types of modules that may be created on a physical machine may be classified in advance, such as low, medium, and high levels; before creating a module on a physical machine, the priority of the module may be determined according to the type of the module to be created and the correspondence between the pre-classified types and priorities.

[0048] The memory specification of a module is used to reflect the memory size occupied by the module. The memory specification can refer to the memory specification required or requested by the module, that is, the memory specification can reflect the memory size requested by the module.

[0049] The currently available page table cache resources are used to characterize the number and types of page table entries currently allowed to be cached on the physical machine. The TLB cache resources available on the physical machine are generally limited. For example, in the TLB module of the IOMMU on a certain platform, there are 8 page table entries for 1GB, 128 page table entries for 2MB, and 256 page table entries for 4KB. The currently available page table cache resources refer to the page table cache resources currently available on the physical machine. The total page table cache resources minus the used page table cache resources are the currently available page table cache resources.

[0050] In step S120, a page table of a first page size is configured for the module, where the first page size is set based on attribute information and / or page table cache resources.

[0051] The first page size refers to the page size used when the virtual address space of the module is divided into multiple pages. A page table corresponding to the module can be generated based on the first page size. Among them, the virtual address space of the module is divided into multiple pages, and the page size of the page is the same as the first page size. The specific generation process of the page table is a mature technology in this field, and reference can be made to the above and the description of the prior art. Generally speaking, when the virtual address space is fixed, the larger the first page size, the fewer the number of divided pages, the fewer the number of levels of the page table hierarchy, and the higher the query efficiency when the TLB misses.

[0052] By setting the page size used when dividing pages, the size of the memory mapping range represented by each page table entry in the page table cache of the module can be configured. The memory mapping range represented by each page table entry in the page table cache is positively correlated with the first page size. The larger the first page size, the larger the memory range corresponding to the mapping relationship between the virtual address and the physical address represented by each page table entry in the page table cache, the more page address mapping relationships that the TLB can store, and the higher the hit rate when querying the TLB.

[0053] The present disclosure can select a suitable page size from a plurality of preset page sizes as the page size used when configuring a page table for a module. For example, a page size can be selected from the three levels of page sizes of 4KB, 2MB, and 1GB as the first page size.

[0054] When determining the page size of the module's page table, the following principles can be followed: when the currently available page table cache resources are sufficient, the size of the first page size can be positively correlated with the module's priority and / or memory specifications; when the currently available page cache resources are limited, a suitable page size can be selected from the currently available page cache resources based on the module's priority and / or memory specifications.

[0055] For example, if the module's memory specification is large (100GB) and its priority is high, the maximum page size (such as 1GB) can be selected from the currently available page cache resources to configure the module page table; if the module's memory specification is large (100GB) but its priority is low, the next-level page size (such as 2MB) can be selected from the currently available page cache resources to configure the module page table; if the module's memory specification is small and its priority is low, the smallest page size (such as 4KB) can be selected from the currently available page cache resources to configure the module page table.

[0056] As an example, based on the number of categories of available page sizes, a predetermined number of levels can be divided, with each level corresponding to a category of page size; based on the memory specifications and priorities of the modules, the level to which the module belongs is determined; if there is an available page size within the level, the page size of the level is selected as the first page size; if there is no available page size within the level, the page size of the next level below the level is selected as the first page size.

[0057] The module's attribute information can characterize the module's demand for page table cache. By configuring a page table of the corresponding page size for the module based on the module's attribute information and / or currently available TLB resources before creating the module on the physical machine, not only can the balanced allocation of TLBs for modules created on the physical machine be achieved while fully considering the module's characteristics and system available resources, but TLB resources can also be reasonably allocated when the module is first created.

[0058] Furthermore, as mentioned above, when frequent cross-page access occurs, the TLB performance of configuring a large page (such as 1GB) for a module is not necessarily better than a smaller page size (such as 2MB). For this reason, the present disclosure proposes that during the operation of the module created on the physical machine, the performance of the module's page table cache can be evaluated in real time, and based on the evaluation results, it is determined whether the page size of the page in the module's page table needs to be adjusted.

[0059] The TLB performance can be monitored, such as monitoring the miss of the TLB. The performance of the page table cache of the module is evaluated based on the monitoring results. For the miss of the TLB of the traditional CPU (such as the number of misses), it can be tracked based on the PMU (Performance Monitor Unit) built into the CPU. However, for the miss of IOTLB, some CPUs do not support the statistics of TLB misses.

[0060] To this end, the present disclosure proposes a new TLB performance monitoring method. When the CPU does not support statistics on TLB misses, the statistics on IOTLB misses can be implemented based on the TLB performance monitoring method of the present disclosure.

[0061] First, a bitmap corresponding to the module can be generated. The bitmap is composed of multiple bits, each bit in the bitmap corresponds to one of the multiple pages into which the module's virtual address space is divided, and the order of each bit in the bitmap is the same as the order of the page corresponding to the bit in the multiple pages. That is, the bits in the bitmap and the multiple pages can correspond one to one in order. Specifically, the multiple pages into which the module's virtual address space is divided are multiple pages with consecutive addresses, and the bitmap is also composed of multiple bits with order. The first bit in the bitmap corresponds to the first page of the multiple pages with consecutive addresses, and the second bit in the bitmap corresponds to the second page of the multiple pages with consecutive addresses, and so on.

[0062] During the operation of the module, the page corresponding to the virtual address accessed by the address access operation (such as DMA operation) initiated for the virtual address space of the module within a predetermined time range can be recorded. Based on the recording result, the bit corresponding to the recorded page in the bitmap is set, and setting means setting the value of the bit to 1.

[0063] The bit settings in the bitmap can characterize the pages involved and not involved in the address access operation within a predetermined time range, and two adjacent bits in the bitmap correspond to two consecutive address pages. Therefore, according to the bit settings in the bitmap, it can be determined whether there are frequent cross-page access operations. Frequent cross-domain access usually leads to TLB misses. Therefore, according to the bit settings in the bitmap, the performance of the page table cache can be evaluated.

[0064] For example, if there are multiple bits with a value of 1 in the bitmap, and there are one or more bits with a value of 0 between two adjacent bits with a value of 1, it can be determined that there are frequent cross-page accesses, the possibility of TLB miss is high, and the performance is low; conversely, if there is no 0 bit between two adjacent bits with a value of 1 in the bitmap, it can be determined that there are no frequent cross-page accesses, the possibility of TLB miss is low, and the performance is high.

[0065] As an example, the page corresponding to the virtual address accessed by the address access operation (such as DMA operation) initiated for the virtual address space of the module within a predetermined time range can be recorded periodically, and a bitmap can be maintained for the recording results of each cycle, so that it can be judged in real time whether there is frequent cross-domain access, and then further infer whether there is a TLBmiss and the severity of the TLBmiss based on the judgment result. Optionally, it can also be combined with the setting of the bits in multiple bitmaps to judge whether there is frequent cross-domain access.

[0066] If the evaluation results show that the performance of the page table cache is poor (e.g., the miss rate of the TLB is higher than a certain threshold), the page table configured for the module can be dynamically adjusted, such as adjusting the previously set first page size to the second page size. When making specific adjustments, the adjustment upward or downward can be determined based on the size of the currently set first page size, the setting of the bit in the bitmap, and the currently available page table cache resources.

[0067] For example, when the current first page size is a large page (such as 1GB) and the setting of the bit in the bitmap indicates that there are frequent cross-page accesses, the first page size can be adjusted downward to the second page size (such as 2MB). This not only improves TLB performance, but also releases large page resources.

[0068] For another example, when the current first page size is small (such as 4KB), and the setting in the bitmap indicates that there are frequent cross-page accesses but the memory range spanned is not too high (such as 1MB or tens of KB), the first page size (4KB) can be adjusted upward to the second page size (such as 2MB).

[0069] According to the second page size, the address space can be re-divided, the page table level of the page table can be re-determined based on the division result (ie, the number of pages), and the page table corresponding to the module can be regenerated based on the re-determined page table level.

[0070] Taking the server as a virtual machine as an example, the address access operation can be a direct memory access operation (i.e., DMA operation) initiated by a device directly connected to the virtual machine. The virtual machine runs in a virtualization program (such as a qemu process). The IOMMU page table update process for the virtual machine is as follows: create a new virtualization program; stop the operation of the virtual machine and the operation of the device directly connected to the virtual machine; redetermine the page table level of the page table according to the second page size; generate a page table corresponding to the module in the new virtualization program according to the redetermined page table level; remap the memory of the old virtualization program to the new virtualization program; resume the operation of the virtual machine and the operation of the device directly connected to the virtual machine.

[0071] Application Examples

[0072] As systems become more integrated, two trends are introduced:

[0073] 1) The memory of a system is getting bigger and bigger; TB-level system configuration has become the mainstream configuration of cloud computing center servers;

[0074] 2) The requirements for system reuse and separation capabilities are getting higher and higher, especially with the large-scale use of containers. The granularity of traditional device allocation through VF (Virtual Function) and PF (Physical Function) is too coarse.

[0075] In order to achieve more precise allocation of devices according to queues, a new device queue virtualization technology Scalable IOV came into being. The large-scale use of Scalable IOV has brought a huge impact on the IOMMU page table system.

[0076] Because the device address space of Scalabe IOV is allocated according to the queue, in extreme cases, each device queue will have its own address space. In IOMMU, the number of available TLB caches is limited. For example, in the TLB module of IOMMU on a certain platform, there are 8 page table entries for 1GB, 128 page table entries for 2M, and 256 page table entries for 4KB. In general, using 1GB large page table entries will improve system performance. It can reduce the number of IOTLB misses and the number of page table levels that need to be traversed after a miss. If there are 64 1GB page table address spaces on the system, directly using 1GB large pages may not necessarily bring corresponding performance improvements. Multiple address spaces conflict with each other between IOTLB table entries, which will have a fatal impact on system performance. For example, if a maximum of 64 1GB table entries are saved, when an address outside the 64GB is accessed, a new page table entry using 1GB large pages will be generated, and one of the original 64 page table entries must be eliminated. If the amount of such elimination is large, it will have a greater impact on performance, because only when it is hit can the performance be improved.

[0077] This embodiment mainly aims at IOMMU TLB interference in scalable IOV scenarios, and proposes an optimization solution that can reduce IOTLB misses and improve system performance from a system level, including the following three parts.

[0078] 1. Tracking of IOTLB access model

[0079] The CPU has a built-in PMU to track the number of TLB misses, but there is currently no corresponding tracking mechanism for IOTLB misses. This embodiment proposes that the device can record the operation range of DMA initiated by the device to make an estimate of the IOTLB miss situation.

[0080] Figure 2 A schematic diagram of the tracking principle of the IOTLB access model according to an embodiment of the present disclosure is shown.

[0081] First, you can apply for a block memory in the system to record the memory range accessed by the device DMA operation. Figure 3 As shown, the memory range accessed by the device DMA operation can be recorded in the bitmap. Specifically, the address space of the entire system can be divided according to the page size of the IOTLB used (4KB / 2MB / 1GB), and each 4KB or 2MB or 1GB corresponds to a bit in the bitmap. Each page of the virtual machine's address space can be marked by finding a page in the bitmap.

[0082] like Figure 2 As shown, a DMA tracking engine can be set for the virtual machine. When the device that directly connects to the virtual machine initiates a DMA operation, the IOVA address corresponding to the operation will be sent to the DMA tracking engine. The DMA tracking engine will check whether the bit in the bitmap corresponding to the page involved in the IOVA is set. If not, it will set it (that is, set 0 to 1), so as to achieve the purpose of recording memory access. Using a bitmap can reduce the amount of data to be processed later.

[0083] Optionally, the system may also directly record the IOVA for subsequent processing, which can improve the accuracy of subsequent system analysis.

[0084] 2. IOTLBmiss analysis

[0085] Usually, for performance reasons, a page table with a page size of 1GB is constructed first for IOMMU mapping. However, the number of entries for a 1GB large page is limited. When a virtual machine (VM) frequently performs DMA operations across 1GB large pages, or when multiple virtual machines perform DMA operations simultaneously, the IOTLB will be frequently swapped in and out, causing serious degradation of the VM's performance.

[0086] By periodically tracking the IOTLB access model of the virtual machine, the possible IOTLB miss in the current system can be estimated. For example, if a system has a 100GB VM and 10 2GB VMs, the IO of the 10 1GB VMs may cause serious interference to the IO performance of the 100GB VM.

[0087] From the IOTLB access model tracking mentioned in the first part, if further confirmed, the 100G VM has frequent cross-page memory access per unit time, such as within 100ms, the memory access range spans 2GB, and the number of accesses is as high as 10,000 times, and these 1GB VMs also have intensive IO access, such as a total of 50,000 IO accesses initiated within 100ms. Combined with the test data, assuming that these 100,000 times occur evenly, it can be basically confirmed that each IO operation of the 100GB VM will trigger an IOTLB miss. Assuming that each IOTLB miss will introduce a delay of 300ns, this will cause a performance drop of nearly 10% to the virtual machine compared to the case of no miss.

[0088] 3. IOTLBmiss optimization

[0089] When there are 4KB / 2MB / 1GB IOTLB resources on the system at the same time, although 1GB generally has the best performance, when all devices use 1GB large pages at the same time, the system performance may not necessarily reach the best, but will form competition. So on the one hand, when the system starts to map IOMMU, it should be allocated well, such as using 1GB large pages for virtual machines with large memory, and 2MB or 4KB for machines with smaller memory. But in fact, the access model of each VM itself is always uncertain. For example, the aforementioned 100GB VM may access 40GB across memory every 100ms, and because there are fewer 1GB large page table entries, it may have better performance using 2MB large pages. Therefore, it is necessary to have the ability to adjust and plan the IOMMU page table.

[0090] Similarly, in the scalable IOV scenario, each device queue is assigned to a separate virtual machine, and there may be thousands of queues on a physical machine. At this time, if everyone uses 1GB at the same time, the TLB cache will be frequently swapped in and out, eventually causing a serious deterioration in system performance.

[0091] 4. IOMMU page table adjustment plan

[0092] a) Resource creation management plan:

[0093] By estimating the VMs that may be created on the physical machine, we try to allocate IOTLB resources appropriately when the VM is first created.

[0094] 1. Mainly refer to the following factors: VM priority, VM memory specification, and current TLB allocation;

[0095] 2. VM priority, that is, the types of VMs that may be created on the physical machine are classified. VMs with higher priorities are allocated large page TLB resources first;

[0096] 3. VM memory specifications. The larger the VM memory specifications, the more obvious the performance enhancement brought by large page TLB.

[0097] 4. Current IOTLB allocation status.

[0098] Specific allocation plans include:

[0099] i. VMs with VM memory greater than or equal to the first threshold (e.g., 16 GB), or VM memory equal to the total memory of the physical machine where the VM is located is greater than or equal to the second threshold (e.g., 5%), will be allocated 1 GB VMs first;

[0100] ii. The system's TLB is currently divided into three levels: 4KB, 2MB, and 1GB. The VM priority can also be divided into three levels from low to high. When the results of step i and step ii are inconsistent, the average of the two results is taken.

[0101] b) Dynamic resource adjustment:

[0102] Obviously, static resource allocation has errors when the system is actually used. Therefore, it is also necessary to have the ability to dynamically adjust resources. Based on the tracking results of the IOTLB access model, IOTLBmiss monitoring can be performed. According to the IOTLBmiss monitoring results, the possible miss rates of 4KB, 2MB, and 1GB page table entries can be analyzed. When the miss results are too high, dynamic adjustments can be made upward or downward based on the actual system conditions. For example, it is found that the IOTLBmiss situation at a certain level exceeds expectations, such as the number of misses of the first-level page is far more than 512 times the number of misses of the higher-level page. At this time, dynamic adjustments are required.

[0103] It should be noted that when the hardware is updated and iterated and gradually has statistical capabilities such as IOTLB miss, some inputs in this embodiment can be optimized. For example, the IOTLB miss situation can be analyzed based on the statistical results of the hardware for IOTLB miss, and dynamic resource adjustment can be performed based on the IOTLB miss situation.

[0104] Dynamic adjustment can be achieved through the hot upgrade of virtual machines. The dynamic adjustment process is briefly described as follows:

[0105] i. Create a new virtualization program:

[0106] ii. Pause the virtual machine;

[0107] iii. Stop the virtual machine device;

[0108] iv. Map the IOMMU page table in the new virtualization program according to the required number of page table levels;

[0109] v. Remap the memory of the virtualization program to the new virtualization program;

[0110] vi. Restore virtual machines and equipment operations at one time;

[0111] In summary, this embodiment breaks the traditional understanding of TLB, that is, the larger the page size, the higher the TLB performance, and balances the system's page table configuration through reasonable detection and analysis to improve the capabilities of the entire system. Specifically, this embodiment proposes a new page table allocation scheme, that is, balancing the allocation of TLB resources while fully considering the characteristics of the virtual machine and the available resources of the system; when IOTLB MISS cannot be counted, an IOTLB miss evaluation scheme is proposed; an IOMMU page table update method is proposed; in addition, this embodiment is not only applicable to IOTLB, but also to TLB.

[0112] The page table cache optimization method disclosed in the present invention may also be implemented as a page table cache optimization device. Figure 4 The schematic diagram of the structure of the page table cache optimization device according to one embodiment of the present disclosure is shown. Among them, the functional unit of the page table cache optimization device can be implemented by hardware, software or a combination of hardware and software that implements the principles of the present disclosure. It can be understood by those skilled in the art that Figure 4 The functional units described may be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein may support any possible combination, division, or further limitation of the functional units described herein.

[0113] The following is a brief description of the functional units that the page table cache optimization device may have and the operations that each functional unit may perform. For the details involved, please refer to the relevant description above and will not be repeated here.

[0114] See also Figure 4 The page table cache optimization device 400 includes an acquisition module 410 and a configuration module 420 .

[0115] The acquisition module 410 is used to obtain the module's attribute information and / or currently available page table cache resources before creating the module on the physical machine. The attribute information is information related to the module's demand for the page table cache. The attribute information may include, but is not limited to, the module's priority and / or memory specifications. The configuration module 420 is used to configure a page table of a first page size for the module, and the first page size is set based on the attribute information and / or the page table cache resources.

[0116] The virtual address space of the module is divided into multiple pages, and the page size of the page is the same as the first page size. The page table cache optimization device 400 may also include a bitmap generation module, a recording module, a setting module and an evaluation module. The bitmap generation module is used to generate a bitmap corresponding to the module, and each bit in the bitmap corresponds to a page in multiple pages; the recording module is used to record the page corresponding to the virtual address accessed by the address access operation initiated for the virtual address space of the module within a predetermined time range during the operation of the module; the setting module is used to set the bit in the bitmap corresponding to the recorded page; the evaluation module is used to evaluate the performance of the page table cache according to the setting of the bit in the bitmap. The configuration module 420 can also adjust the first page size to the second page size based on the evaluation result.

[0117] The page table cache optimization device 400 may further include a page table level determination module for redetermining the page table level of the page table according to the second page size; and a page table generation module for regenerating a page table corresponding to the module based on the redetermined page table level.

[0118] The page table updating method disclosed in the present invention may also be implemented as a page table updating device. Figure 5 FIG. 1 is a schematic diagram showing a structure of a page table update device according to an embodiment of the present disclosure. The functional unit of the page table update device can be implemented by hardware, software, or a combination of hardware and software that implements the principles of the present disclosure. It can be understood by those skilled in the art that Figure 5 The functional units described may be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein may support any possible combination, division, or further limitation of the functional units described herein.

[0119] The following is a brief description of the functional units that the page table update device may have and the operations that each functional unit may perform. For the details involved, please refer to the relevant description above and will not be repeated here.

[0120] See also Figure 5 , the page table updating device 500 includes an evaluation module 510 and a determination module 520 .

[0121] The evaluation module 510 is used to evaluate the performance of the page table cache of the module during the operation of the module created on the physical machine; the determination module 520 is used to determine whether the page size of the page in the page table of the module needs to be adjusted based on the evaluation result.

[0122] The evaluation module 510 may include a bitmap generation module, a recording module, a setting module, and an evaluation submodule. The bitmap generation module is used to generate a bitmap corresponding to the module, and each bit in the bitmap corresponds to a page in the page table. The recording module is used to record the page corresponding to the virtual address accessed by the address access operation initiated for the virtual address space of the module within a predetermined time range during the operation of the module. The setting module is used to set the bit corresponding to the recorded page in the bitmap. The evaluation submodule is used to evaluate the performance of the page table cache according to the setting of the bit in the bitmap.

[0123] The page table updating device 500 may further include an adjustment module, a page table level determination module, and a page table generation module. The adjustment module is used to adjust the page size from the first page size to the second page size if it is determined that the page size of a page in the page table of the module needs to be adjusted. The page table level determination module is used to redetermine the page table level of the page table according to the second page size. The page table generation module is used to regenerate the page table corresponding to the module based on the redetermined page table level.

[0124] Figure 6 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned page table cache optimization method or page table update method according to an embodiment of the present disclosure is shown.

[0125] See also Figure 6 , the computing device 600 includes a memory 610 and a processor 620 .

[0126] The processor 620 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 620 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 620 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0127] The memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 620 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 610 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0128] The memory 610 stores executable codes. When the executable codes are processed by the processor 620 , the processor 620 can execute the page table cache optimization method or the page table update method mentioned above.

[0129] The page table cache optimization method, page table update method, apparatus, and device according to the present disclosure have been described above in detail with reference to the accompanying drawings.

[0130] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0131] Alternatively, the present disclosure may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present disclosure.

[0132] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0133] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A page table cache optimization method, comprising: Before creating a module that uses a virtual address to access memory on a physical machine, obtaining attribute information of the module and / or currently available page table cache resources, wherein the attribute information is information related to the degree of demand of the module for the page table cache; A page table of a first page size is configured for the module, where the first page size is set based on the attribute information and / or the page table cache resource.

2. The method according to claim 1, wherein: The attribute information includes priority and / or memory specification.

3. The method according to claim 2, wherein: The size of the first page size is positively correlated with the priority and / or memory specification of the module.

4. The method according to claim 1, wherein: The virtual address space of the module is divided into a plurality of pages, the page size of the pages is the same as the first page size, and the method further comprises: generating a bitmap corresponding to the module, each bit in the bitmap corresponding to one of the plurality of pages; During the operation of the module, the page corresponding to the virtual address accessed by the address access operation initiated for the virtual address space of the module within a predetermined time range is recorded; Setting a bit in the bitmap corresponding to the recorded page; The performance of the page table cache is evaluated based on the settings of the bits in the bitmap.

5. The method according to claim 4, further comprising: The first page size is adjusted to a second page size based on the evaluation result.

6. The method according to claim 5, further comprising: re-determining the number of page table levels of the page table according to the second page size; Based on the re-determined page table level number, a page table corresponding to the module is regenerated.

7. The method according to claim 5, wherein: The module is a virtual machine, the address access operation is initiated by a device that directly passes through the virtual machine, and the virtual machine runs in a virtualization program. The method also includes: Create new virtualization programs; Stop the operation of the virtual machine and the operation of the device directly connected to the virtual machine; re-determining the number of page table levels of the page table according to the second page size; Generating a page table corresponding to the module in a new virtualization program according to the re-determined page table level; Remap the memory of the old virtualizer to the new virtualizer; Resume the operation of the virtual machine and the devices that are passed through the virtual machine.

8. A page table updating method, comprising: During the operation of the module created on the physical machine, evaluating the performance of the page table cache of the module; Determining whether to adjust the page size of a page in the page table of the module based on the evaluation result; If it is determined that the page size of the page in the page table of the module needs to be adjusted, the page size is adjusted from the first page size to the second page size; the first page size is set before the module is created on the physical machine based on the attribute information of the module and / or the page table cache resource, and the attribute information is information related to the degree of demand of the module for the page table cache; re-determining the number of page table levels of the page table according to the second page size; Based on the re-determined page table level number, a page table corresponding to the module is regenerated.

9. The method according to claim 8, wherein: The steps of evaluating the performance of the page table cache of the module include: generating a bitmap corresponding to the module, each bit in the bitmap corresponding to a page in the page table; During the operation of the module, a page corresponding to a virtual memory address accessed by a direct memory access operation initiated for a virtual address space of the module within a predetermined time range is recorded; Setting a bit in the bitmap corresponding to the recorded page; The performance of the page table cache is evaluated based on the settings of the bits in the bitmap.

10. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 9.

11. A computer program product, comprising executable codes, which, when executed by a processor of an electronic device, cause the processor to perform the method according to any one of claims 1 to 9.

12. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 9.

Images