Virtual machine resource isolation strategy based on MPAM assistance

Abstract

The invention discloses a virtual machine resource isolation strategy based on MPAM assistance, which is based on MPAM hardware mechanism assistance of ARMv8.4, and is matched with a page coloring software mechanism to isolate fine-grained memory shared resources such as cache and memory bandwidth in a hybrid system for deploying a virtual machine monitor. The method comprises the following steps: firstly, defining a concept of fine-grained memory shared resources; and then providing a software and hardware isolation mechanism aiming at the fine-grained memory shared resources, wherein the software and hardware isolation mechanism mainly comprises a software mechanism based on page coloring, a coloring bit determination algorithm, a hardware isolation mechanism based on MPAM and a resource isolation and memory bandwidth control mechanism under the combination of the software and hardware mechanisms. According to the virtual machine resource isolation strategy, rich resource isolation strategies are provided, the problem that the performance of the key virtual machine is reduced due to resource competition can be effectively solved, the system resource utilization rate is increased, and the optimization capability of a virtualization technology on a hybrid key system is fully exerted.

Description

Technical Field

[0001] The present invention relates to the field of operating systems and virtualization, and in particular to an isolation strategy for fine-grained memory shared resources in a virtualized hybrid deployment scenario, and in particular to a virtual machine resource isolation strategy based on MPAM assistance. Background Art

[0002] With the increase in the number of processor cores and the continuous upgrading of the architecture of computing platforms, computer processing performance has been greatly improved. Computer systems have gradually evolved from simple devices that rely on single-chip microcomputers or microcontrollers to complex devices that can focus on different needs such as real-time needs, security needs, and business needs. To this day, in order to save power consumption, cost, and volume of computer systems, multiple subsystems with different functional requirements or critical levels are integrated on the same hardware platform, which is academically called a mixed critical system. How to ensure the safe partitioning of the system is the most important issue that needs to be considered in a mixed critical system.

[0003] In the virtualization scenario, virtualization technology implicitly provides software support for partitioning on different operating system construction environments. Virtualization technology provides a software running at a privileged level - Virtual Machine Monitor (VMM, or Hypervisor), and the virtual machine monitor abstracts hardware resources to ensure that different critical and isolated application environments can run on the same physical platform. These operating environments are called virtual machines (VM).

[0004] On the one hand, the virtual machine monitor can partition the resources of the physical platform through software and hardware support, and because it is at a higher privilege level, it can adopt different strategies for virtual machines of different criticalities to optimize mixed-criticality systems.

[0005] Physical resources include computing resources (CPU), storage resources (memory), and device resources (I / O devices) at a macro level. The virtual machine monitor can manage these physical resources in a unified manner, abstract and allocate these resources, and provide isolated partitions for upper-level virtual machines. The design of the static partitioning hypervisor (SPH) can better isolate the above resources to meet the needs of mixed-criticality systems. Static partitioning refers to the practice of allocating all platform resources to different subsystems when the system is built or initialized. Depending on the hardware primitives, static partitioning can be concretized in a variety of forms, and virtualization provides strong encapsulation guarantees and flexible configuration, making it a natural helper of the static partitioning architecture.

[0006] In addition, for some implicitly shared and conceptually finer-grained physical resources, such as cache, DRAM bank, and memory bandwidth, existing open source static partitioned virtual machine monitors ignore their isolation. Considering some existing studies, Frank et al. partitioned instructions and data by modifying the compiler to achieve cache partitioning; Farzad et al. partitioned cache and DRAM by hardware modification of MMU and TLB. Both modifications are for special software and hardware, which are difficult to modify and lack certain versatility. Kloda et al. used Jailhouse to allocate part of the non-overlapping cache to different upper-level virtual machines during memory partitioning to achieve isolation of mixed-criticality system cache. This method has certain reference significance, but it only isolates cache, and the software implementation brings certain limitations, such as limiting the second-stage large page mapping and limiting the allocation of continuous addresses. The above resource isolation implemented by software has more or less some shortcomings or introduces additional performance overhead. The possible reason is that these microarchitecture resources ignored the problem of multi-system sharing at the beginning of hardware design, so the latest designed hardware mechanism is needed to complete the corresponding operation.

[0007] Hardware isolation means, such as the Resource Director Technology (RDT) proposed by Intel, are not intended for virtualization. Intel-cmt-cat is a user-mode tool for RDT. To realize virtualization functions, a mechanism similar to vCAT is required; ARMv8.4 introduced the Memory System Resource Partitioning and Monitoring (MPAM) function, such as the Openeuler4.19.36 kernel, which controls the business flow based on the Kunpeng 920 chip, but this isolation is still completed at the kernel level and does not meet the requirements of a mixed-criticality system; Rehm et al.'s introduction to the MPAM mechanism only stays at the possible assistance to mixed-criticality systems, and has not been actually implemented or experimentally verified. These hardware mechanisms only provide a high-level specification and interface, but the use of these interfaces alone lacks flexibility and is often inseparable from the flexible configuration and algorithm strategies of software.

[0008] In summary, for resource isolation and partitioning in mixed-criticality systems, a set of methods and mechanisms that combine software and hardware mechanisms are needed to ensure strong resource isolation while maintaining a certain degree of flexibility. Summary of the invention

[0009] The purpose of the present invention is to provide an isolation and management strategy that can fully cover and support flexible adjustment for fine-grained memory shared resources such as cache, DRAM Bank and memory bandwidth through a combination of software and hardware mechanisms assisted by MPAM.

[0010] The technical solution of the present invention is: A virtual machine resource isolation strategy based on MPAM assistance includes the following steps: The first step is to use the mapping relationship between the allocation of cache and DRAM banks and the locations of memory addresses when allocating physical address space to virtual machines, and allocate striped memory resources so that different virtual machines have independent cache resources and DRAM Bank partitions; The second step is to allocate physical memory around the color position, including: maintaining a global two-dimensional array with the color position as the index and the size of the physical page as the minimum granularity, wherein the memory is managed in the same color position in a preset structure, and the preset structure refers to a structure that includes the color position, the physical memory start address, the number of memory blocks in the same color position, and whether the color block is available, and is used to identify the relationship between the physical memory and the color position; The third step is to implement virtual resource control under the MPAM mechanism, including: during the virtual machine configuration phase, the virtual machine monitor divides the system's memory shared resources such as the last-level cache and memory bandwidth according to the MPAM configuration information; then, each memory access instance is assigned a corresponding PARTID and PMG identifier, the virtual machine monitor's identifier value is set to 0, and the virtual machine's identifier value is set to VMID + 1; finally, the virtual machine monitor's PARTID and PMG identifiers will be bound to the MPAM2_EL2 values ​​of all CPUs, and the virtual machine's identifier value will be correspondingly bound to the virtual machine CPU, i.e., the MPAM1_EL1 context of the vCPU, and the corresponding value will be updated when the vCPU context is switched; Step 4: The virtual machine monitor sets the bandwidth limit for CPU access to memory by using PMU.

[0011] The beneficial effects of the present invention are: (1) It provides a variety of resource isolation strategies, which can effectively reduce the performance degradation of key virtual machines caused by resource competition; (2) Resource control technology assisted by MPAM can effectively integrate system resources, improve system resource utilization, and give full play to the optimization capabilities of virtualization technology for hybrid critical systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1This is a schematic diagram of virtual machine management based on MPAM assistance; Figure 2 It is the core data structure of page coloring Figure 3 This is the pseudo code for confirming the DRAM Bank mapping relationship; Figure 4 It is the dynamic adjustment process of memory bandwidth under the combination of software and hardware mechanisms; DETAILED DESCRIPTION

[0013] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0014] The basic principle of the resource isolation strategy of the present invention is: with the assistance of the MPAM (Memory System Resource Partitioning and Monitoring) hardware mechanism, combined with the page coloring software mechanism, the memory resources in the system are managed to effectively alleviate the problem of fine-grained memory shared resource competition.

[0015] The virtual machine resource isolation strategy based on MPAM assistance includes four steps. Among them, the first and second steps are pure software implementations. The first step involves discovering the coloring number, and the second step involves how to modify the memory allocation. The third and fourth steps are pure MPAM hardware mechanisms, and the combination of the above software mechanisms. Therefore, in the following steps, the software implementation is introduced first, and then the isolation strategy after the hardware mechanism is introduced.

[0016] The first step is to use the mapping relationship between the allocation of cache and DRAM storage banks and the locations of memory addresses when allocating physical address space to virtual machines, and allocate striped memory resources so that different virtual machines have independent cache resources and DRAM bank partitions.

[0017] This step includes: calculating the shading locations corresponding to the cache and DRAM Bank in the system.

[0018] The shading position corresponding to the cache is calculated as follows: First calculate the number of cache shading bits: , in, is the size of the last-level cache; is the number of paths in the group associativity, such as 4-path group associativity ; is the size of the physical page. The above values ​​are all powers of 2, and the number of shader bits obtained is also a power of 2.

[0019] Secondly, according to the number of shading bits, the position starting from bit 12 of the physical address is selected as the shading position.

[0020] Regarding the shading position of the DRAM Bank, the experiment will test the latency of each address bit for memory access. According to the characteristics of DRAM, when the address related to the shading bit is hit, the latency will increase significantly. Figure 3 The algorithm involved in this calculation process is shown.

[0021] In this step, the cache and DRAM Bank are isolated based on the software mechanism of page coloring. The basic idea is that there is a mapping relationship between the allocation of cache and DRAM Bank and the location of the memory address. According to this mapping relationship, when the virtual machine allocates physical address space, these locations are used as coloring to allocate striped memory resources, so that different virtual machines have independent cache resources and DRAM Bank partitions. The focus of this strategy is to confirm the mapping relationship between the allocation of fine-grained shared resources and physical address encoding, and to organize the striped memory that the virtual machine can use through a certain data structure.

[0022] The software resource isolation mechanism based on page coloring in this step can deploy applications in any system and provide effective isolation.

[0023] The second step is to allocate physical memory around the color position, including: using the color position as the index and the size of the physical page as the minimum granularity, maintaining a global two-dimensional array, which can be named MEM_REGION_BY_COLOR. In the global two-dimensional data, the memory in the same color position is managed in the form of a ColorMemRegion structure, such as Figure 2 As shown, the ColorMemRegion structure refers to a structure that contains information such as the shading position, the starting address of the physical memory, the number of memory blocks at the same shading position, and whether the shading block is available (i.e., whether it has been allocated to other virtual machines). It is used to identify the relationship between physical memory and shading numbers. The shading block refers to the shading space represented by the ColorMemRegion structure. The memory includes virtual machine memory and fine-grained memory. For the virtual machine memory, each virtual machine will be configured with its corresponding shading number. During the initialization phase of the virtual machine, the virtual machine memory is allocated according to the shading number. At this time, because each shading area is separated from each other and there is no overlap, the self-designed variables in the ColorMemRegion structure are used during allocation to ensure that each shading area is uniquely assigned to a certain virtual machine, so that the isolation of virtual machine memory and fine-grained memory shared resources can be completed. The self-designed variable can be named available, which is used to indicate whether the memory represented by the structure has been allocated.

[0024] The shading bit discovery mechanism of this step can provide a universal shading bit discovery algorithm, which can effectively confirm the shading bit information of the cache and DRAM Bank on any physical platform.

[0025] The third step is to implement the control of virtual resources under the MPAM mechanism. Figure 1 As shown in the figure, during the virtual machine configuration phase, the virtual machine monitor will divide the system's memory shared resources such as the last-level cache and memory bandwidth according to the MPAM configuration information; then, the corresponding PARTID and PMG (Performance Monitoring Group) identifiers are assigned to each memory access instance, the virtual machine monitor identifier value is set to 0, and the virtual machine identifier value is set to VMID + 1; finally, the virtual machine monitor's PARTID and PMG identifiers will be bound to the MPAM2_EL2 values ​​of all CPUs, and the virtual machine identifier value will be bound to the MPAM1_EL1 context of the virtual machine CPU, i.e., the vCPU, and the corresponding values ​​will be updated when the vCPU context switches. MPAM1_EL1 and MPAM2_EL2 are the information (i.e., PARTID and PMG) generated for memory requests at different exception levels in the Advanced RISC Machine (ARM) architecture.

[0026] According to this step, the hardware mechanism based on MPAM can complete resource partitioning with virtual machines as the smallest granularity under the control of the virtual machine monitor, thereby ensuring resource flow partitioning of virtual machines of different criticality levels in the mixed criticality system.

[0027] Use the isolation interface provided based on the MPAM specification, including: a. Partitioning of cache: The cache is divided into several parts through hardware mechanism, and these parts are allocated according to PARTID. PARTID is an identifier under MPAM mechanism, which can be understood and called partition number.

[0028] After MPAM divides the cache, each PARTID will divide the corresponding partition bitmap to identify which parts of the system have control rights. If the corresponding bitmap is set to 1, the access rights of the corresponding part are granted to the corresponding PARTID. Partitions can be assigned to different PARTIDs exclusively or shared, that is, the virtual machine monitor can manage the use of the virtual machine's cache and divide the cache according to the criticality of the virtualization system.

[0029] b. Minimum and maximum partitions of memory bandwidth: The memory controller is required to identify memory access traffic. When the memory bandwidth is less than the minimum partition limit corresponding to PARTID, the traffic will be processed first; otherwise, if it is greater than the maximum partition limit corresponding to PARTID, the traffic will be suppressed. By limiting the memory bandwidth of other non-critical virtual machines, the memory bandwidth of critical virtual machines is protected from intrusion.

[0030] Step 4. The virtual machine monitor sets the upper limit of the bandwidth for vCPU to access memory by using the performance monitoring unit (PMU). The PMU event is selected as direct memory access (Data Memory Access) by setting the PMXEVTYPER_EL0 register (Performance Monitors Selected Event Type Register, the register for PMU to select monitoring events), and the access quota of different vCPU memory bandwidth is set by setting the PMXEVCNTR_EL0 register (Performance Monitors Selected Event Count Register, the counter for observing the number of times the system sends related events after the above register selects the event). If the number of memory accesses is too many, an overflow will occur and an interrupt will be generated. The interrupt is treated as a reserved interrupt and processed by the virtual machine monitor, and the corresponding bandwidth (the access quota of the vCPU memory bandwidth) is adjusted according to the real-time data (memory bandwidth usage data) monitored by the MPAM hardware mechanism.

[0031] According to this step, the bandwidth reservation mechanism implemented by PMU and software with the assistance of MPAM can effectively manage the bandwidth resources in the system and ensure that the memory access bandwidth of real-time critical virtual machines is not affected by bandwidth competition.

[0032] Through the third and fourth steps, the software and hardware mechanisms are combined with MPAM to isolate resources. The software and hardware combination mechanism first complements each other and enriches the isolation strategy. Secondly, the monitoring mechanism of MPAM is used to monitor the usage of cache and memory bandwidth, thereby providing dynamic adjustment capabilities; for virtualized systems, such as Figure 4As shown in the figure, this paper selects PMU to set the memory access event count of different virtual machines. When the memory access counter overflows, an interrupt is triggered. This is used as an opportunity for dynamic adjustment. Each virtual machine is assigned PMG as a memory resource usage label. The usage of memory bandwidth at this time is known through MPAM. Different virtual machines are processed separately: when the bandwidth of the real-time critical virtual machine is less than the minimum bandwidth lower limit set by MPAM, the value of the minimum bandwidth lower limit is slightly increased, and the operation of other non-critical virtual machines is blocked to ensure that the bandwidth of the performance-critical virtual machine can be quickly supplemented to the minimum bandwidth lower limit; non-real-time critical virtual machines adjust their bandwidth upper limit according to the observed value to reserve more bandwidth for real-time critical virtual machines.

Claims

1. A virtual machine resource isolation strategy based on MPAM assistance, characterized in that: The following steps are involved: The first step is to use the mapping relationship between the allocation of cache and DRAM banks and the locations of memory addresses when allocating physical address space to virtual machines, and use these locations as coloring to allocate striped memory resources, so that different virtual machines have independent cache resources and DRAM Bank partitions; The second step is to allocate physical memory around the color position, including: maintaining a global two-dimensional array with the color position as the index and the size of the physical page as the minimum granularity, wherein the memory is managed in the same color position in a preset structure, and the preset structure refers to a structure that includes the color position, the physical memory start address, the number of memory blocks in the same color position, and whether the color block is available, and is used to identify the relationship between the physical memory and the color position; The third step is to implement virtual resource control under the MPAM mechanism, including: the virtual machine monitor divides the system's memory shared resources according to the MPAM configuration information during the virtual machine configuration phase; then, the corresponding PARTID and PMG identifiers are assigned to each memory access instance, the virtual machine monitor's identifier value is set to 0, and the virtual machine's identifier value is set to VMID + 1; finally, the virtual machine monitor's PARTID and PMG identifiers will be bound to the MPAM2_EL2 values ​​of all CPUs, and the virtual machine's identifier value will be correspondingly bound to the virtual machine CPU, i.e., the MPAM1_EL1 context of the vCPU, and the corresponding value will be updated when the vCPU context is switched; Step 4: The virtual machine monitor sets the bandwidth limit for vCPU access to memory by using PMU.

2. The MPAM-assisted virtual machine resource isolation strategy according to claim 1 is characterized in that: The first step includes: calculating the shading location corresponding to the cache and DRAM Bank in the system, wherein the shading location corresponding to the cache is calculated as follows: First calculate the number of cache shading bits: ， in is the size of the last-level cache; is the number of paths in the group association, is the size of the physical page; Secondly, according to the number of shading bits, the position starting from bit 12 of the physical address is selected as the shading position.

3. The MPAM-assisted virtual machine resource isolation strategy according to claim 1 is characterized in that: The fourth step includes: By setting the PMXEVTYPER_EL0 register, the PMU event is selected as direct memory access, and by setting the PMXEVCNTR_EL0 register, the access quota of different vCPU memory bandwidths is set. If the number of memory accesses is too many, an overflow will occur and an interrupt will be generated. The interrupt will be treated as a reserved interrupt and processed by the virtual machine monitor. The access quota of the vCPU memory bandwidth is adjusted according to the real-time data monitored by the MPAM hardware mechanism.

Images