Virtual machine computing resource dynamic allocation method and system based on PCIe card and storage medium

Abstract

The invention discloses a virtual machine computing resource dynamic allocation method and system based on a PCIe card and a storage medium, and belongs to the field of virtualization technology and hardware resource management.The method comprises the steps that the computing unit resource utilization rate of the PCIe card is monitored in real time through a hardware abstraction layer; dynamically generating a resource allocation scheme based on an improved auction algorithm, wherein the algorithm optimizes a resource quota through a time attenuation factor and a resource reuse penalty term; utilizing a programmable register remapping technology to distribute independent VF equipment for the virtual machine and establish a direct access channel; realizing interruption isolation between virtual functions through an interruption redirection controller; and during resource redistribution, atomic migration of a calculation task is completed by adopting a check point migration mechanism, and the atomic migration comprises context preloading and PCIe link layer retraining. According to the method, the problems of hardware resource distribution rigidness, insufficient isolation and migration service interruption in the prior art are solved, and fine-grained control and service lossless switching of the heterogeneous computing units are realized.

Description

Technical Field

[0001] The present invention relates to the field of virtualization technology and computer hardware resource management, and in particular to a method, system and storage medium for dynamically allocating virtual machine computing resources based on a PCIe card, which is particularly suitable for realizing efficient utilization and real-time adjustment of a virtual machine's computing unit of a PCIe device through hardware-level resource partitioning, dynamic register remapping and interrupt isolation technology. Background Art

[0002] In cloud computing and high-performance computing scenarios, virtual machines (VMs) have an increasing demand for dynamic allocation of hardware resources. Traditional virtual machine resource allocation technology mainly implements the scheduling of common resources such as CPU and memory through the software layer (such as Hypervisor), but for heterogeneous computing units such as PCIe devices (such as GPU, FPGA, AI acceleration card), there are the following problems: 1. Rigid allocation of hardware resources: Existing technologies mostly use static partitioning or software simulation (such as SR-IOV basic functions), which cannot dynamically adjust physical resource quotas (such as video memory bandwidth, DMA channels) according to virtual machine loads, resulting in low resource utilization. For example, when multiple virtual machines compete for GPU computing units, fixed resource allocation may cause performance bottlenecks or idle resources.

[0003] 2. Insufficient interrupt and isolation: Traditional interrupt processing relies on virtual machine monitor (VMM) forwarding, which increases latency and makes it difficult to achieve strict isolation between virtual functions (VFs). In addition, the IOMMU-based address translation mechanism does not support fine-grained partitioning of PCIe device on-chip caches (such as L2 caches), which may cause security vulnerabilities or data conflicts.

[0004] 3. Business interruption during migration: When adjusting PCIe device resources, existing resource migration technologies (such as hot plugging) need to suspend virtual machine tasks and reinitialize device contexts, making it difficult to ensure business continuity.

[0005] In response to the above problems, the prior art has proposed some improvement plans. For example, Chinese patent CN106020933A proposes a resource scheduling method based on ultra-lightweight virtual machines, which realizes the dynamic allocation and recovery of virtual machine resources through resource prediction modules and task queuing modules. The core of this method is to use linear fitting and quadratic fitting algorithms to predict resource utilization and prioritize them according to the SLA agreement. However, this method lacks hardware adaptation and only performs software-layer scheduling for general resources such as CPU and memory, without considering the resource isolation and dynamic allocation requirements of heterogeneous PCIe device computing units such as GPU and FPGA; interrupt processing is inefficient and relies on the virtual machine monitor (VMM) to forward interrupt requests, resulting in insufficient isolation between virtual functions (VFs), making it difficult to meet the requirements of high real-time scenarios; the migration mechanism is rigid, resource recovery relies on static threshold judgment, and the migration process requires pausing virtual machine tasks to reinitialize the context, making it impossible to achieve lossless business switching.

[0006] Chinese patent CN106681839 discloses a method for dynamic allocation of elastic computing, which selects overloaded physical hosts through a greedy algorithm and migrates virtual machines to low-load nodes based on a hotspot balancing formula. Its advantage is that it reduces contention for CPU and memory resources. It does not solve the problem of dynamic partitioning of physical resources of PCIe device computing units (such as AI acceleration cards) in a single host, and lacks hardware-level partitioning support for caches (such as L2 caches) on PCIe cards, which can easily cause data conflicts.

[0007] Chinese patent CN114625544A discloses a memory balloon management method and device based on dynamic allocation of virtual machine business load. The method dynamically adjusts memory allocation by monitoring the virtual machine business throughput rate and uses the memory bubble mechanism to achieve resource reuse. However, the method has a narrow scope of application and only optimizes memory resources. It does not cover the dynamic allocation requirements of PCIe device computing units (such as GPU memory bandwidth and DMA channels), and cannot achieve register-level remapping and interrupt isolation of PCIe configuration space.

[0008] Therefore, there is an urgent need for a virtual machine resource allocation solution that can combine hardware-level resource partitioning, dynamic load awareness and lossless migration to improve the utilization, isolation and business continuity of heterogeneous computing resources. Summary of the invention

[0009] In view of the existing technical problems, the purpose of the present invention is to provide a method, system and storage medium for dynamic allocation of virtual machine computing resources based on PCIe card, which realizes efficient utilization and real-time adjustment of PCIe device computing units by virtual machines through hardware-level resource partitioning, dynamic register remapping and interrupt isolation technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions: A PCIe card-based virtual machine computing resource dynamic allocation method, comprising: Monitoring resource usage of each computing unit on at least one PCIe card in real time through a hardware abstraction layer, wherein the PCIe card is connected to a host machine through a PCIe switch, wherein the resource usage includes computing unit temperature, cache occupancy, and video memory bandwidth; Based on the preset virtual machine resource allocation strategy and the monitored dynamic load data, a real-time allocation scheme including resource quotas and isolation parameters is generated, wherein the resource allocation strategy includes priority weights, physical resource segmentation granularity, and service quality level, and the real-time allocation scheme is dynamically adjusted through an improved auction algorithm, wherein the improvement includes the introduction of a time decay factor and a resource reuse penalty item; The target PCIe card's computing unit is partitioned at the hardware level through programmable register remapping technology, an independent VF device is allocated to each virtual machine, and a direct access channel with the virtual machine's operating system is established; Dynamically adjust the BAR space mapping relationship corresponding to each virtual machine according to the real-time allocation plan, and implement interrupt isolation of virtual functions through the interrupt redirection controller; When resources are reallocated, the business continuity of the virtual machine is maintained through the checkpoint migration mechanism, including preloading context data in the target computing unit, using a double-buffer switching strategy to complete the atomic migration of computing tasks, and maintaining the stability of the physical connection through PCIe link layer retraining.

[0011] Preferably, the hardware abstraction layer includes: A type adapter module for identifying the computing unit architecture type of a heterogeneous PCIe device, including a GPU stream processor, an FPGA logic unit, and an AI accelerator tensor core; The virtualization agent module is deployed in the embedded controller of the PCIe card and is used to execute resource monitoring instructions and return extended monitoring data, wherein the extended monitoring data includes computing unit temperature and cache occupancy rate.

[0012] Preferably, the generation process of the real-time allocation scheme specifically includes: Matching a preset resource template according to the business type of the virtual machine, wherein the template defines the computing unit occupancy ratio, the video memory bandwidth threshold, and the DMA channel allocation rule; When resource contention is detected, resource quotas are dynamically adjusted based on the improved auction algorithm, wherein a time decay factor is used to reduce the weight of historical resource occupancy, and a resource reuse penalty term is used to limit repeated allocation of the same physical resource.

[0013] Preferably, the hardware-level resource partitioning operation includes: Allocate independent VF devices to each virtual machine through PCIe configuration space rewriting technology; Divide the on-chip cache of the PCIe card into address windows and establish a dedicated L2 cache partition for the virtual machine.

[0014] Furthermore, the method further comprises: SR-IOV technology is used to create multiple physical function instances, and each instance is configured with an independent TC queue to isolate traffic priorities.

[0015] The present invention also provides a resource allocation system for implementing the method, comprising: The dynamic perception module is integrated in the PCIe root complex of the host machine to collect real-time performance counter data of each VF device and analyze the TLP packet statistical traffic characteristics of the PCIe protocol layer; A policy engine module, deployed on the out-of-band management controller, includes a resource prediction model based on reinforcement learning and a dynamic priority calculator, wherein the dynamic priority calculator is used to generate a weight coefficient in combination with a virtual machine SLA agreement and historical load data; A reconfiguration execution module, including a programmable clock generator and a power management unit, for adjusting the operating frequency and supply voltage of the computing unit; The security isolation module adopts an IOMMU-based address translation protection mechanism, configures a dynamically updated access control list, and implements DMA access authentication by whitelist filtering of write operations on the PCIe configuration space.

[0016] Preferably, the dynamic perception module further includes: The temperature adaptive unit is directly connected to the thermal sensor on the PCIe card and triggers resource migration when overheating is detected.

[0017] Preferably, the safety isolation module further implements: The address space of the virtual machine is isolated through PCIe ATS (Address Translation Services).

[0018] The present invention also provides a computer-readable storage medium storing instructions for executing the method, wherein when the instructions are executed by a processor: Hot-swap support for PCIe resources is implemented by modifying the device pass-through module of the virtual machine monitor, including dynamic loading and unloading of VF device drivers; The integrated resource topology discovery algorithm automatically identifies the computing resource distribution by traversing the port registers of the multi-level PCIe switch architecture.

[0019] Due to the adoption of the above scheme, the beneficial effects of the present invention are as follows: 1. Improved resource utilization: Through real-time monitoring of the hardware abstraction layer and an improved auction algorithm, fine-grained dynamic allocation of PCIe card computing units is achieved, and resource reuse rate is increased by more than 30%; 2. Enhanced isolation: Hardware-level partitioning and interrupt redirection technologies reduce the resource contention rate between virtual machines to less than 5%, ensuring the service quality of high-priority tasks; 3. Lossless business switching: The checkpoint migration mechanism reduces the business interruption time of the resource reallocation process to milliseconds, meeting the needs of highly sensitive scenarios such as financial transactions and real-time rendering; 4. Compatibility and scalability: Supports multiple heterogeneous PCIe devices such as GPU, FPGA, AI acceleration card, and can be adapted to different virtualization platforms (such as KVM, Xen). DETAILED DESCRIPTION

[0020] The technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Embodiment 1: Implementation of a dynamic resource allocation method based on a PCIe card Step 1: Hardware Abstraction Layer Real-time Monitoring The hardware abstraction layer is deployed on the host machine to connect the NVIDIA A100 GPU card and the Xilinx Alveo U280 FPGA card. The type adapter module identifies the GPU card's stream multiprocessor (SM) architecture (containing 108 computing units) and the FPGA's programmable logic block (CLB) array (containing 1,228,800 logic units). The virtualization agent module is embedded in the GPU card's DPU (BlueField-2) and collects resource data every 50ms, including stream processor temperature (≤85℃), L2 cache occupancy (≤90%), and video memory bandwidth (550GB / s). By parsing the TLP packets of the PCIe 4.0 protocol layer, the DMA channel traffic characteristics of the virtual machine VM1 are counted (such as 1,200 requests per second and an average data volume of 4KB). When the GPU temperature exceeds 80℃, resource migration to the FPGA card is triggered.

[0022] Step 2: Dynamic resource allocation strategy generation Match the preset resource template for AI inference virtual machines, define the computing unit occupancy ratio of 40%, the memory bandwidth threshold of 200GB / s, and the number of DMA channels of 2. When it is detected that the memory bandwidth utilization reaches the threshold, the improved auction algorithm is used to dynamically adjust the resource quota. Set the time decay factor α=0.85 and the resource reuse penalty term β=0.3, according to the formula:

[0023] in, Q(i,k): resource quota of virtual machine i in the kth cycle (such as GPU computing unit occupancy, video memory bandwidth, etc.); α: time decay factor (value range 0<α<1), used to reduce the weight of historical resource occupancy; R(i): basic quota based on the SLA agreement (e.g. AI inference VMs are allocated 40% computing units by default); β: resource reuse penalty term (value range β>0), used to limit the frequency of repeated allocation of the same physical resource; Reuse times: The cumulative number of times the target resource (such as a video memory channel) is repeatedly allocated within the most recent T time window.

[0024] Calculate the resource quota of virtual machine VM1, increase the GPU computing unit occupancy from 35% to 42%, and adjust the video memory bandwidth from 180 GB / s to 210 GB / s.

[0025] Step 3: Hardware-level resource partitioning and isolation Through PCIe configuration space rewriting technology (offset address 0×200), an independent virtual function (VF1) is allocated to VM1, and the BAR space mapping is set to 0×10000000-0×18000000 (128MB video memory). Two physical function instances (PF0 and PF1) are created using SR-IOV technology, and the traffic category (TC) queue weight is configured as TC1:TC0=3:1. The interrupt redirection controller (Intel VT-d) maps VF1's interrupt request to VM1's exclusive interrupt vector (IRQ 48), and sets a DMA access whitelist through IOMMU to limit VF1 to access only its allocated BAR space.

[0026] Step 4: Checkpoint Migration and Business Continuity Guarantee When resources are reallocated, the CUDA kernel state (register value, instruction pointer) of VF1 is saved to the checkpoint file and preloaded to the L2 cache partition of the target PF1. A double buffer switching strategy is adopted to complete the atomic migration of computing tasks within 1ms, during which the packet loss rate is ≤0.005%. PCIe link layer retraining is performed (LTSSM state machine jumps to Recovery state) to ensure that the link rate is stable at 16GT / s and the bit error rate is ≤1×10 -12 .

[0027] Example 2: Resource allocation system deployment The dynamic perception module is integrated into the PCIe root complex (Intel Xeon Scalable processor) of the host machine, and collects the instruction cycle (IPC=1.2) and cache hit rate (92%) of the VF device through the performance monitoring unit (PMU). The cross-device monitoring unit parses the traffic matrix of the TLP packet and generates a virtual machine resource usage report. The policy engine module is deployed on the out-of-band management controller (BMC), runs a resource prediction model based on reinforcement learning, inputs historical load data (time window 5 minutes) and virtual machine SLA level, and outputs dynamic priority weights (such as VM1:VM2=0.8:0.5). The security isolation module enables the PCIe address translation service (ATS), maps the customer physical address (GPA 0×2000) of VM1 to the system physical address (SPA 0×10000000), and configures the access control list (ACL) to restrict out-of-bounds operations.

[0028] Example 3: Storage medium instruction execution Modify the device passthrough module of QEMU-KVM and add hot-plug callback functions to support loading the VF driver (nvidia-vf.ko) of NVIDIA GPU at runtime. Execute the resource topology discovery algorithm, traverse the port register (Port 0×1C00) of the PCIe switch (MicrochipSwitchtec PSX), and identify the multi-level topology structure (such as root port → Switch0 → GPU card and FPGA card). Output the resource distribution map for the scheduler to ensure that the virtual machine allocates computing units on demand.

[0029] Experimental verification and effect data: 1. Resource utilization comparison is shown in Table 1: Table 1 index Traditional SR-IOV solution The present invention GPU compute unit utilization 68% 91% Memory bandwidth contention rate 22% 5% Migration downtime 20ms 0.5ms .

[0030] 2. Isolation Verification When VF1 performs high-load matrix operations (1000x1000) and VF2 runs real-time inference tasks, the standard deviation of VF2's latency fluctuation drops from 18% to 2%, verifying the effectiveness of interrupt isolation.

Claims

1. A method for dynamically allocating virtual machine computing resources based on a PCIe card, characterized in that: include: Monitoring resource usage of each computing unit on at least one PCIe card in real time through a hardware abstraction layer, wherein the PCIe card is connected to a host machine through a PCIe switch, wherein the resource usage includes computing unit temperature, cache occupancy, and video memory bandwidth; Based on the preset virtual machine resource allocation strategy and the monitored dynamic load data, a real-time allocation scheme including resource quotas and isolation parameters is generated, wherein the resource allocation strategy includes priority weights, physical resource segmentation granularity, and service quality level, and the real-time allocation scheme is dynamically adjusted through an improved auction algorithm, wherein the improvement includes the introduction of a time decay factor and a resource reuse penalty item; The target PCIe card's computing unit is partitioned at the hardware level through programmable register remapping technology, an independent VF device is allocated to each virtual machine, and a direct access channel with the virtual machine's operating system is established; Dynamically adjust the BAR space mapping relationship corresponding to each virtual machine according to the real-time allocation plan, and implement interrupt isolation of virtual functions through the interrupt redirection controller; When resources are reallocated, the business continuity of the virtual machine is maintained through the checkpoint migration mechanism, including preloading context data in the target computing unit, using a double-buffer switching strategy to complete the atomic migration of computing tasks, and maintaining the stability of the physical connection through PCIe link layer retraining.

2. The method according to claim 1, characterized in that The hardware abstraction layer includes: A type adapter module for identifying the computing unit architecture type of a heterogeneous PCIe device, including a GPU stream processor, an FPGA logic unit, and an AI accelerator tensor core; The virtualization agent module is deployed in the embedded controller of the PCIe card and is used to execute resource monitoring instructions and return extended monitoring data, wherein the extended monitoring data includes computing unit temperature and cache occupancy rate.

3. The method according to claim 1, characterized in that The generation process of the real-time allocation scheme specifically includes: Matching a preset resource template according to the business type of the virtual machine, wherein the template defines the computing unit occupancy ratio, the video memory bandwidth threshold, and the DMA channel allocation rule; When resource contention is detected, resource quotas are dynamically adjusted based on the improved auction algorithm, wherein a time decay factor is used to reduce the weight of historical resource occupancy, and a resource reuse penalty term is used to limit repeated allocation of the same physical resource.

4. The method according to claim 1, characterized in that: The hardware-level resource partitioning operation includes: Allocate independent VF devices to each virtual machine through PCIe configuration space rewriting technology; Divide the on-chip cache of the PCIe card into address windows and establish a dedicated L2 cache partition for the virtual machine.

5. The method according to claim 4, characterized in that Also includes: SR-IOV technology is used to create multiple physical function instances, and each instance is configured with an independent TC queue to isolate traffic priorities.

6. A resource allocation system for implementing the method according to any one of claims 1 to 5, characterized in that: include: The dynamic perception module is integrated in the PCIe root complex of the host machine to collect real-time performance counter data of each VF device and analyze the TLP packet statistical traffic characteristics of the PCIe protocol layer; A policy engine module, deployed on the out-of-band management controller, includes a resource prediction model based on reinforcement learning and a dynamic priority calculator, wherein the dynamic priority calculator is used to generate a weight coefficient in combination with a virtual machine SLA agreement and historical load data; A reconfiguration execution module, including a programmable clock generator and a power management unit, for adjusting the operating frequency and supply voltage of the computing unit; The security isolation module adopts an IOMMU-based address translation protection mechanism, configures a dynamically updated access control list, and implements DMA access authentication by whitelist filtering of write operations on the PCIe configuration space.

7. The system according to claim 6, characterized in that The dynamic perception module also includes: The temperature adaptive unit is directly connected to the thermal sensor on the PCIe card and triggers resource migration when overheating is detected.

8. The system according to claim 6, characterized in that The safety isolation module further implements: The address space of the virtual machine is isolated through PCIe ATS (Address Translation Services).

9. A computer-readable storage medium storing instructions for executing the method according to any one of claims 1 to 5, characterized in that: When the instruction is executed by the processor: Hot-swap support for PCIe resources is implemented by modifying the device pass-through module of the virtual machine monitor, including dynamic loading and unloading of VF device drivers; The integrated resource topology discovery algorithm automatically identifies the computing resource distribution by traversing the port registers of the multi-level PCIe switch architecture.

Images