Method and system for pci device address persistence in virtualized environment

Abstract

This application provides a method and system for persistent PCI device addresses in a virtualized environment. The method involves creating an address inheritance QoS policy for the device to be persisted, writing PCI address metadata, and adding a minimum bandwidth rule to bind NUMA nodes and physical network cards. The scheduler reserves the corresponding PCI address slot according to this rule, reads the metadata, and restores the PCI address in the reserved slot, thus achieving address persistence. This application reuses the platform's native QoS policy capabilities without modifying the core architecture, enabling cross-virtual machine PCI address inheritance, avoiding address conflicts, ensuring the stable operation of services relying on fixed PCI addresses, and adapting to large-scale cloud environment deployments.

Description

Technical Field

[0001] This invention belongs to the field of virtualization technology application technology, and in particular relates to a method and system for persistent PCI device addresses in a virtualization environment. Background Technology

[0002] In virtualized environments, the inability to inherit PCI addresses in cross-virtual machine scenarios such as virtual machine cloning, image reconstruction, and cold migration has been a long-standing technical problem. In existing technologies, PCI address metadata is only bound to the instance object of the source virtual machine. When a new virtual machine generates a new instance object, it cannot inherit the PCI address information of the source virtual machine; the virtualization platform must dynamically allocate a new PCI address. For applications that rely on fixed PCI addresses, dynamically allocated addresses can cause applications to fail to recognize the device, leading to service interruptions. Even if the source virtual machine's PCI address is manually configured in the new virtual machine, that PCI address on the target node may already be occupied by another virtual machine or process, causing the new virtual machine to fail to start. Existing solutions require modifying the virtualization platform's database structure to add metadata transmission fields or modifying the scheduler logic to add cluster-level address pool management. These solutions intrude into the platform's core architecture, affecting platform compatibility and stability, and are unsuitable for deployment requirements in large-scale cloud environments. Summary of the Invention

[0003] This application provides a method and system for persistent PCI device addresses in a virtualized environment. Without modifying the core architecture of the virtualization platform, it can achieve cross-instance inheritance of PCI addresses in cross-virtual machine scenarios such as virtual machine cloning, cold migration, and image reconstruction, and ensure that the recovered PCI addresses are available and do not conflict on the target node.

[0004] This application discloses a method for persistent PCI device addresses in a virtualized environment, including: Create an address inheritance QoS policy for the device to be persisted, write the device's PCI address metadata into the address inheritance QoS policy, and add a minimum bandwidth rule to the address inheritance QoS policy. The minimum bandwidth rule is bound to the NUMA node and physical network card corresponding to the PCI address. According to the minimum bandwidth rule, the scheduler reserves a slot for the corresponding PCI address for the device; reads the PCI address metadata in the address inheritance QoS policy, restores the device's PCI address in the reserved slot, and persists the PCI device address.

[0005] Optionally, create an address inheritance QoS policy for the device to be persisted, including: Obtain the device's PCI address information and device identifier; Encode the PCI address information and device identifier to generate PCI address metadata; Create an address inheritance QoS policy and write the PCI address metadata into the description field of the address inheritance QoS policy.

[0006] Optionally, the PCI address information and device identifier are encoded to generate PCI address metadata, including: Organize PCI address information and device identifiers into structured data; The structured data is compressed; the compressed structured data is then encoded to generate PCI address metadata. Calculate the checksum of the PCI address metadata and write the checksum into the rule field of the address inheritance QoS policy.

[0007] Optionally, creating an address inheritance QoS policy for the device to be persisted also includes: Check whether the detection device has been bound to the service QoS policy; If the device is already bound to a service QoS policy, create an address-inherited QoS sub-policy and set the parent policy of the address-inherited QoS sub-policy to the service QoS policy. Bind the address inheritance QoS sub-policy to the device.

[0008] Optionally, create an address inheritance QoS policy for the device to be persisted, including: When the device is a network interface card (NIC) device, call the network service API to create a network-side address inheritance QoS policy; When the device is a disk device, call the storage service API to create a storage-side address that inherits the QoS policy.

[0009] Optionally, based on the minimum bandwidth rule, the scheduler reserves slots for corresponding PCI addresses for the devices, including: The scheduler identifies the address bound to the device and inherits the minimum bandwidth rule in the QoS policy; Filter out compute nodes that contain the corresponding physical network interface card and have available slots in the NUMA node; On the selected computing nodes, slots corresponding to PCI addresses are reserved for the devices, and the slots are marked as occupied.

[0010] Optionally, read the PCI address metadata from the address inheritance QoS policy, restore the device's PCI address in the reserved slot, and persist the PCI device address, including: Read the PCI address metadata and checksum from the address inheritance QoS policy; The PCI address metadata is verified, and the PCI address information is obtained after the verification is successful. Create a device address configuration object and write the PCI address information into the device address configuration object; The device address configuration object is prewritten into the device's configuration properties, overriding the dynamic address allocation logic.

[0011] Optionally, the method further includes: When a device is destroyed, the corresponding address inheriting the QoS policy is automatically deleted. When the PCI address of a device changes, the PCI address metadata and minimum bandwidth rules in the address inheritance QoS policy are updated synchronously.

[0012] Optionally, the method further includes: Create a shared address template QoS policy, and write the PCI address templates of a batch of devices into the shared address template QoS policy; Bind a shared address template QoS policy to multiple devices; Multiple devices read the PCI address template from the shared address template QoS policy and restore the PCI address of the corresponding serial number device.

[0013] This application also discloses a PCI device address persistence system in a virtualized environment, including: The policy creation module is used to create an address inheritance QoS policy for the device to be persisted, write the device's PCI address metadata into the address inheritance QoS policy, and add a minimum bandwidth rule to the address inheritance QoS policy. The minimum bandwidth rule is bound to the NUMA node and physical network card corresponding to the PCI address. The scheduling reservation module is used to reserve slots for corresponding PCI addresses for devices according to the minimum bandwidth rule; the address recovery module is used to read the PCI address metadata in the address inheritance QoS policy and restore the PCI address of the device in the reserved slot to persist the PCI device address.

[0014] As can be seen from the above technical solution, this application first carries PCI address metadata through address inheritance QoS policy. Utilizing the strong binding of QoS policy with devices and its automatic synchronization with device cloning or migration, it ensures that PCI address metadata can be transferred from the source virtual machine to the new virtual machine. Simultaneously, it configures a minimum bandwidth rule for binding the corresponding NUMA node and physical network card to the QoS policy. When the scheduler selects a node for the new virtual machine, it first identifies this rule, filtering out compute nodes with corresponding physical network cards and NUMA nodes with available PCI address slots. Then, it reserves the target PCI address slot on the selected node to prevent it from being occupied by other processes. Finally, when the new virtual machine starts, it directly reads the metadata in the QoS policy and completes the original PCI address configuration in the reserved slot without modifying the core architecture of the virtualization platform. This achieves cross-virtual machine inheritance of PCI addresses and avoids conflicts during address recovery, ensuring the continuous and stable operation of services relying on fixed PCI addresses and adapting to the deployment needs of large-scale cloud environments. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is one of the flowcharts of a PCI device address persistence method in a virtualized environment according to the embodiments of this application; Figure 2 This is a flowchart illustrating the creation of an address inheritance QoS policy for a device to be persisted in a PCI device address persistence method in a virtualized environment, as described in an embodiment of this application. Figure 3 This is a flowchart illustrating a method for persistent PCI device addresses in a virtualized environment, as described in this application, which encodes PCI address information and device identifiers to generate PCI address metadata. Figure 4 This is a flowchart illustrating a PCI device address persistence method in a virtualized environment, as described in this application embodiment, which creates an address that inherits a QoS policy for the device to be persisted. Figure 5 This is the second flowchart of a PCI device address persistence method in a virtualized environment according to an embodiment of this application, which creates an address inheritance QoS policy for the device to be persisted. Figure 6 This is a flowchart illustrating a method for persistent PCI device addresses in a virtualized environment, as described in this application embodiment, in which the scheduler reserves corresponding PCI address slots for devices based on minimum bandwidth rules. Figure 7 This is a flowchart illustrating a method for persisting PCI device addresses in a virtualized environment, as described in this application embodiment. The method reads PCI address metadata from the address inheritance QoS policy, restores the device's PCI address in a reserved slot, and thus persists the PCI device address. Figure 8 This is the second flowchart of a PCI device address persistence method in a virtualized environment according to an embodiment of this application; Figure 9 This is the third flowchart of a PCI device address persistence method in a virtualized environment according to an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a PCI device address persistence system in a virtualized environment, provided as an embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] In this embodiment, the virtualization environment includes multiple computing nodes, each running a virtualization management service for managing the lifecycle and device configuration of virtual machines. The computing nodes communicate with a control node via a network. The control node runs network services, storage services, and scheduling services, used to manage network resources, storage resources, and virtual machine scheduling, respectively.

[0019] Taking the virtual machine cloning scenario of running DPDK applications as an example, in existing technologies, the PCI address of the source virtual machine's network interface card (NIC) is stored in the instance metadata of the source virtual machine. When a user initiates a cloning request, the control node creates a new virtual machine instance and generates new instance metadata. This new instance metadata does not contain the source virtual machine's PCI address information. When the new virtual machine starts, the virtualization management service dynamically allocates a new PCI address to the NIC. Since the DPDK application is bound to the source virtual machine's PCI address during compilation or configuration, the DPDK application in the new virtual machine cannot find the NIC corresponding to the new address, causing the application to fail to start. To solve this problem, operations personnel usually need to manually modify the new virtual machine's configuration file to set the PCI address to the source virtual machine's address. However, in large-scale cloud environments, manual configuration is inefficient and error-prone. Furthermore, even if the original address is manually configured, it may already be occupied by another virtual machine or process on the target node, causing the new virtual machine to fail to start. Existing solutions require modifying the control node's database structure to add a field for storing cross-virtual machine address metadata, or modifying the scheduling service logic to add a cluster-level address pool management module. These modifications will intrude on the platform's core architecture, affecting its compatibility and stability, and making it impossible to smoothly upgrade to the new version of the virtualization platform.

[0020] Based on this, embodiments of this application provide a method for persistent PCI device addresses in a virtualized environment, such as... Figure 1 As shown, it includes: S100: Create an address inheritance QoS policy for the device to be persisted, write the device's PCI address metadata into the address inheritance QoS policy, and add a minimum bandwidth rule to the address inheritance QoS policy. The minimum bandwidth rule is bound to the NUMA node and physical network card corresponding to the PCI address. S200: According to the minimum bandwidth rule, the scheduler reserves a slot for the corresponding PCI address for the device; reads the PCI address metadata in the address inheritance QoS policy, restores the device's PCI address in the reserved slot, and persists the PCI device address.

[0021] This application provides a method for persistent PCI device addresses in a virtualized environment. First, PCI address metadata is carried through an address inheritance QoS policy. Leveraging the strong binding between the QoS policy and the device, and its automatic synchronization during device cloning or migration, the method ensures that PCI address metadata can be transferred from the source virtual machine to the new virtual machine. Simultaneously, a minimum bandwidth rule is configured for the QoS policy, binding the corresponding NUMA node and physical network interface card (NIC). When the scheduler selects a node for the new virtual machine, it first identifies this rule, filtering out compute nodes with the corresponding physical NIC and available PCI address slots on the NUMA nodes. Then, a slot for the target PCI address is reserved on the selected node to prevent it from being occupied by other processes. Finally, when the new virtual machine starts, it directly reads the metadata from the QoS policy and completes the original PCI address configuration in the reserved slot without modifying the core architecture of the virtualization platform. This achieves cross-virtual machine inheritance of PCI addresses and avoids conflicts during address recovery, ensuring the continuous and stable operation of services relying on fixed PCI addresses and adapting to the deployment requirements of large-scale cloud environments.

[0022] It should be noted that the core concept of this application is to reuse the metadata carrying capacity of the native QoS policy of the virtualization platform and the scheduling binding capacity of the minimum bandwidth rule. By creating an address inheritance QoS policy for the device to be persisted, the PCI address metadata of the device is written into the policy. At the same time, a minimum bandwidth rule is added to the address inheritance QoS policy to bind the NUMA node and physical network card corresponding to the PCI address. This triggers the scheduler to reserve the corresponding PCI address slot for the device. Finally, when the new virtual machine starts, the PCI address metadata in the address inheritance QoS policy is read, and the original PCI address of the device is restored in the reserved slot. The entire process can achieve cross-virtual machine persistence of PCI address without modifying the core architecture, API interface and database structure of the platform.

[0023] The embodiments of this application will be described in detail below.

[0024] In this embodiment, the devices to be persisted include network interface cards (NICs) and disk devices. NICs include virtio NICs, SR-IOV virtual function NICs, and PCI pass-through NICs. Disk devices include virtio disks and SCSI disks. Address inheritance QoS policy refers to a QoS policy specifically designed to carry PCI address metadata. Its lifecycle is bound to the lifecycle of the corresponding device, and the address inheritance QoS policy is destroyed synchronously when the device is destroyed. Minimum bandwidth rule refers to the rule in the QoS policy used to guarantee network bandwidth. In this embodiment, the bandwidth value of the minimum bandwidth rule is set to 1kbps. This value is only used to trigger the scheduler's binding logic and will not affect the actual bandwidth allocation of the service. NUMA node refers to a non-uniform memory access node in a compute node. Each NUMA node corresponds to independent CPU, memory, and PCI bus resources. Binding a device to a specific NUMA node can improve service performance. Physical NIC refers to the physical network interface card on a compute node. Each physical NIC contains multiple physical and virtual functions. SR-IOV virtual function NICs are network interfaces virtualized from the physical functions of physical NICs.

[0025] In this embodiment, the step of creating an address-inherited QoS policy for the device to be persisted is executed when the virtual machine starts up for the first time or when the device is first mounted to the virtual machine. The policy creation process is triggered after the virtualization management service completes the PCI address allocation for the device. After the device's PCI address metadata is written into the address-inherited QoS policy, the address metadata is stored in the control node's database along with the QoS policy. When adding a minimum bandwidth rule to the address-inherited QoS policy, the NUMA node and physical network interface card (NIC) corresponding to the rule need to be specified; this information is obtained from the device's PCI address allocation results.

[0026] The process of reserving PCI address slots for devices based on minimum bandwidth rules is performed during virtual machine cloning, image reconstruction, or cold migration. When a user initiates such a request, the scheduling service on the control node selects a suitable compute node for the new virtual machine. When selecting a compute node, the scheduling service identifies the minimum bandwidth rule in the QoS policy inherited by the device's bound address, filters compute nodes based on the NUMA node and physical network interface card information in the rule, and reserves the corresponding PCI address slot on the selected compute node. The reserved slot is marked as occupied and will not be allocated to any other device or process.

[0027] The step of reading the PCI address metadata from the address inheritance QoS policy and restoring the device's PCI address in the reserved slot to persist the PCI device address is executed when the new virtual machine starts. When the new virtual machine is scheduled to the target compute node, the compute node's virtualization management service reads the address inheritance QoS policy bound to the device and extracts the PCI address metadata from it. Based on the PCI address information in the metadata, the virtualization management service configures the device's PCI address in the slot reserved by the scheduler, overriding the default dynamic address allocation logic. After the new virtual machine starts, the device's PCI address is completely consistent with the source virtual machine, and applications that rely on fixed addresses can run normally.

[0028] In an optional implementation, an address inheritance QoS policy is created for the device to be persisted, such as... Figure 2 As shown, it includes: S210: Obtain the device's PCI address information and device identifier; S220: Encode PCI address information and device identifier to generate PCI address metadata; S230: Create an address inheritance QoS policy and write the PCI address metadata into the description field of the address inheritance QoS policy.

[0029] In this embodiment, the step of obtaining the device's PCI address information and device identifier is achieved by intercepting the virtual machine configuration file generated by the virtualization management service. The virtual machine configuration file contains configuration information for all devices, including the device's PCI address, device type, and device identifier. For network interface cards (NICs), the device identifier is a unique identifier for the port; for disk devices, the device identifier is a unique identifier for the volume.

[0030] The step of encoding PCI address information and device identifiers to generate PCI address metadata converts structured address information into a string format suitable for storage in the QoS policy description field. This encoding process reduces the size of the metadata while ensuring its integrity and resolvability.

[0031] When creating an address-inherited QoS policy, the policy name uses a fixed prefix plus device identifier format for easy identification and management. PCI address metadata is written to the description field of the address-inherited QoS policy because the description field is a text field natively supported by QoS policies with no length limit, capable of storing string data of any format without modifying the QoS policy's data structure.

[0032] In an optional embodiment, the PCI address information and device identifier are encoded to generate PCI address metadata, such as... Figure 3 As shown, it includes: S240: Organize PCI address information and device identifier into structured data; S250: Compresses structured data; encodes the compressed structured data to generate PCI address metadata; S260: Calculate the checksum of the PCI address metadata and write the checksum into the rule field of the address inheritance QoS policy.

[0033] In this embodiment, the step of organizing PCI address information and device identifier into structured data uses a key-value pair format. The structured data includes fields such as device type, device identifier, PCI address, bus type, NUMA node, physical network interface card name, version number, and timestamp. The device type field distinguishes between network interface cards and disk devices; the bus type field distinguishes between different bus types such as PCI bus and SCSI bus; the version number field identifies the version of the metadata; and the timestamp field records the creation time of the metadata.

[0034] To enable those skilled in the art to implement this solution without ambiguity, the encoding, compression, and checksum storage mechanisms involved in steps S220-S260 are illustrated below with examples.

[0035] In one specific embodiment, the process of encoding PCI address information and device identifier to generate PCI address metadata in step S220 is as follows: Organizing structured data: First, organize the acquired information into a structured data object in JSON format. For example: json {"device_type":"nic", "device_id":"port-uuid-1234", "pci_address":"0000:3b:00.0", "bus_type":"pci", "numa_node":1, "physical_nic":"ens7f1", "version":"1.0", "timestamp":"2023-10-27T10:00:00Z"} Compression: The above JSON string is losslessly compressed using a common gzip or zlib algorithm, converting it into a byte sequence.

[0036] Encoding: The compressed byte sequence is encoded using the Base64 encoding scheme, converting it into a string composed of ASCII characters. This string is the PCI address metadata described in this application.

[0037] Write: The compressed and Base64 encoded PCI address metadata string is written completely into the description field of the address inheritance QoS policy.

[0038] For the writing and storage of the verification value in step S260, this application adopts a digest-based verification and field reuse scheme to solve the problem of mismatch between the verification value length and the QoS rule field value range. The specific operation is as follows: Calculate the hash digest: Calculate the hash digest value of the Base64 encoded string generated in step 3, for example, using the SHA-256 algorithm.

[0039] Rewrite the rule field: Convert the first few bytes of the calculated hash digest value (e.g., the first 2 bytes, or 16 bits) into an unsigned integer. Then, write this integer value into the specific rule field of the QoS policy.

[0040] For network-side policies, this value will be configured as the tag value for the DSCP tagging rule. It's important to note that the functional semantics of this rule have been redefined from a network service quality tag to a prefix for the address metadata integrity check value. Since the minimum bandwidth rule for trigger slot reservations ensures that devices are bound to specific nodes and network cards, there is usually no further need for actual DSCP tagging. If conflicts arise, they can be resolved by creating a QoS sub-policy for address inheritance. This sub-policy's DSCP rules override the actual service's DSCP rules because the address inheritance policy prioritizes address recovery throughout its lifecycle.

[0041] For storage-side policies, this value will be configured as the value of the read IOPS limit rule. Similarly, its functional semantics are overloaded as a checksum carrier.

[0042] Verification and parsing: During the address recovery phase (S360-S370), the virtualization management service performs the reverse operation: read the Base64 string from the description field -> calculate its hash digest and extract the corresponding length prefix -> compare it with the value read from the rule field -> if they match, continue to decode the Base64 string and then gzip decompress it to restore the original JSON data, and finally extract fields such as pci_address.

[0043] The step of compressing structured data is used to reduce its volume and avoid excessively large amounts of data in the descriptive fields. The compression process employs a common lossless compression algorithm, which can reduce data volume without data loss.

[0044] The step of encoding the compressed structured data involves using a common text encoding algorithm to convert the binary compressed data into text format, making it easier to store in the description field.

[0045] The step of calculating the checksum of PCI address metadata is used to subsequently verify the integrity and correctness of the metadata. The checksum is calculated using a common verification algorithm, capable of detecting corruption or tampering of metadata during storage or transmission. The checksum is written to the rule field of the address inheritance QoS policy because the rule field is a numeric field natively supported by the QoS policy and can store the checksum data, eliminating the need for adding custom fields. For network-side address inheritance QoS policies, the checksum is written to the DSCP tag rule field; for storage-side address inheritance QoS policies, the checksum is written to the read IOPS per second rule field.

[0046] In an optional embodiment, an address inheritance QoS policy is created for the device to be persisted, such as... Figure 4 As shown, it also includes: S270: Check whether the device has been bound to a service QoS policy; S280: If the device is already bound to a service QoS policy, create an address-inherited QoS sub-policy and set the parent policy of the address-inherited QoS sub-policy to the service QoS policy. S290: Bind the address inheritance QoS sub-policy to the device. In this embodiment, the step of detecting whether the device has bound a service QoS policy is implemented by calling the API of the network service or storage service to query the device's QoS policy binding information. A service QoS policy refers to a QoS policy created by a user to control the traffic or IO performance of a service, including service rules such as traffic rate limiting, IO rate limiting, and DSCP marking.

[0047] If the device is already bound to a service QoS policy, an address-inherited QoS sub-policy is created, and the parent policy of the address-inherited QoS sub-policy is set to the service QoS policy. The QoS sub-policy can inherit all rules from the parent policy and can also add its own rules. In this embodiment, the address-inherited QoS sub-policy only includes address metadata and minimum bandwidth rules and will not override any service rules in the parent policy. The service QoS policy has higher priority than the address-inherited QoS sub-policy; when there is a conflict between the two, the rules of the service QoS policy prevail.

[0048] After binding the address inheritance QoS sub-policy to the device, the device will apply the rules of both the service QoS policy and the address inheritance QoS sub-policy simultaneously. The service rules will take effect normally, and the address metadata and minimum bandwidth rules will also function normally without affecting each other.

[0049] In an optional embodiment, an address inheritance QoS policy is created for the device to be persisted, such as... Figure 5 As shown, it includes: S310: When the device is a network card device, call the network service API to create a network-side address inheritance QoS policy; S320: When the device is a disk device, call the storage service API to create a storage-side address that inherits the QoS policy.

[0050] In this embodiment, when the device is a network interface card (NIC) device, a network-side address inheritance QoS policy is created by calling the Network Service API. The Network Service API is an application programming interface provided by the network service on the control node, used to manage network resources and QoS policies. The network-side address inheritance QoS policy is stored in the network service's database and is strongly bound to the NIC port. When the NIC port is cloned, migrated, or rebuilt, the network-side address inheritance QoS policy is automatically synchronized and bound to the new NIC port.

[0051] When the device is a disk device, the storage service API is invoked to create a storage-side address-inherited QoS policy. The storage service API is an application programming interface provided by the storage service on the control node for managing storage resources and QoS policies. The storage-side address-inherited QoS policy is stored in the storage service's database and is strongly bound to the disk volume. When a disk volume is cloned, migrated, or rebuilt, the storage-side address-inherited QoS policy is automatically synchronized and bound to the new disk volume.

[0052] Creating separate QoS policies for network-side and storage-side addresses allows for full utilization of the native QoS policy management capabilities of network and storage services, ensuring that address metadata is transmitted synchronously throughout the device's lifecycle.

[0053] In an optional embodiment, the scheduler reserves a slot for the corresponding PCI address for the device according to the minimum bandwidth rule, such as... Figure 6 As shown, it includes: S330: The scheduler identifies the address bound to the device and inherits the minimum bandwidth rule in the QoS policy; S340: Filter out compute nodes that contain the corresponding physical network card and have available slots in the NUMA node; S350: On the selected computing nodes, reserve slots corresponding to PCI addresses for the devices and mark the slots as occupied.

[0054] In this embodiment, the step of the scheduler identifying the minimum bandwidth rule in the address inheritance QoS policy bound to the device is executed when the scheduling service selects a compute node for the new virtual machine. The scheduling service obtains the QoS policy binding information of all devices of the new virtual machine and filters out the address inheritance QoS policy that contains the minimum bandwidth rule.

[0055] The steps for filtering compute nodes that contain the corresponding physical network interface card (NIC) and have available slots on the NUMA node are as follows: First, filter all compute nodes in the cluster that contain the physical NIC specified in the minimum bandwidth rule. Then, for each filtered compute node, check whether there is an available slot for the corresponding PCI address on the NUMA node specified in the minimum bandwidth rule. A PCI address slot consists of a bus number, slot number, and function number, and each slot is unique on a NUMA node.

[0056] For example, to achieve slot reservation precision from coarse-grained scheduling rules to PCI bus-level precision, this method introduces an independent, globally consistent NUMA node PCI address slot allocation table component at the cluster level. This component can be a lightweight distributed consistent key-value store (such as etcd) or a new table in the scheduler database; this application does not impose restrictions or elaborate on this. Its key consists of compute node ID: NUMA node ID: PCI address (BDF), and the value is the virtual machine ID occupying the slot or a value set to idle. The scheduler workflow is as follows: The scheduling filter plugin parses the target PCI address (e.g., 0000:3b:00.0) from the address inheritance QoS policy; it queries the slot allocation table to confirm that the target slot is currently idle among all eligible candidate compute nodes; once the target compute node is selected, the scheduler initiates an atomic Compare-And-Swap (CAS) operation on the table while making the final scheduling decision, updating the target slot's status from idle to reserved: new virtual machine ID. Only nodes that successfully complete the CAS operation will be ultimately used, thus preventing PCI address conflicts that may occur in concurrent scheduling scenarios. When a virtual machine is destroyed, the slot it occupied is marked as idle again in the table.

[0057] On the selected compute nodes, a slot corresponding to the PCI address is reserved for the device. Marking the slot as occupied involves the scheduling service sending a slot reservation instruction to the selected compute nodes. Upon receiving the instruction, the virtualization management service of the compute node marks the specified PCI slot as occupied, and this slot will not be allocated to any other device or process. The validity period of the slot reservation is consistent with the lifecycle of the new virtual machine; the slot will be automatically released after the new virtual machine is destroyed.

[0058] To more clearly illustrate how the scheduler can reserve PCI address slots without modifying the core code, the steps S330 to S350 above will be elaborated in more detail below.

[0059] In a preferred embodiment of this application, the scheduling service on the control node includes a pluggable scheduling filter plugin (SchedulerFilterPlugin). This plugin is developed in accordance with the native filter interface specification of the scheduling service, without intruding on the core scheduling logic of the scheduling service. The step in step S330, where the scheduler identifies the minimum bandwidth rule in the QoS policy inherited by the address bound to the device, is specifically implemented by this plugin.

[0060] The scheduling filter plugin works as follows: When the scheduling service initiates the process of selecting a target compute node for a new virtual machine, the plugin is invoked and obtains the QoS policy binding information of all devices of the new virtual machine. The plugin filters out policies whose rule fields contain minimum bandwidth rules and whose bandwidth values ​​are specific meaningless flag values ​​(e.g., 1kbps), i.e., the address inheritance QoS policy described in this application.

[0061] After recognizing the address inheritance QoS policy, the plugin executes the following scheduling filtering logic: Metadata parsing: Decode and parse the PCI address metadata from the policy description field, and extract the target PCI bus address (e.g., 0000:3b:00.0) and the bound physical network card identifier.

[0062] Candidate Node Filtering: Traverse all available compute nodes in the cluster to generate a candidate node list. For a compute node to pass the filter, it must simultaneously meet the following two conditions: The compute node has the physical network interface card (NIC) corresponding to the physical NIC identifier installed; and in the PCI bus topology of the target NUMA node affinity-bound to the physical NIC on the compute node, the target PCI bus address slot is currently idle and not occupied by any other virtual machine.

[0063] Slot Status Management: To accurately determine slot status, this embodiment maintains a global PCI address slot status table at the cluster management layer. This status table records the mapping relationship between all PCI bus address slots on each compute node and the virtual machine identifiers occupying them. The update mechanism of the status table is as follows: Reservation: When a new virtual machine is scheduled to a node using this method and a PCI slot is successfully reserved, the scheduler will update the status of the corresponding slot in the status table to occupied and associate it with the virtual machine.

[0064] Release: When a virtual machine is destroyed or migrated to another node, the virtualization management service of the compute node sends a slot release notification to the scheduler, which then updates the status of the corresponding slot in the status table to idle. This slot status table is crucial to ensuring that addresses do not conflict between different virtual machine lifecycles.

[0065] Node Marking and Command Issuance: After the scheduling filter plugin selects all candidate compute nodes that meet the criteria, the scheduling service will choose the node with the most sufficient resources as the target node. Once selected, the scheduling service will execute an atomic operation, which includes: Immediately mark the target PCI slot on the target node in the PCI address slot status table as reserved; Send a virtual machine deployment request carrying a slot reservation instruction to the virtualization management service of the target compute node.

[0066] Through the above process, this solution utilizes the filter plugin extension mechanism natively supported by the scheduler to achieve the awareness and reservation of PCI slots. The entire process does not require modification of the scheduler's core logic, nor does it require the introduction of an independent resource management module, significantly reducing the intrusion into the platform's core architecture.

[0067] In an optional embodiment, the PCI address metadata in the address inheritance QoS policy is read, and the device's PCI address is restored in the reserved slot to persist the PCI device address, such as... Figure 7 As shown, it includes: S360: Reads PCI address metadata and checksum values ​​from the address inheritance QoS policy; S370: Verify the PCI address metadata, and parse the PCI address information after the verification is successful; S380: Create a device address configuration object and write the PCI address information into the device address configuration object; S390: Prewrite the device address configuration object into the device's configuration properties, overriding the dynamic address allocation logic.

[0068] In this embodiment, the step of reading the PCI address metadata and checksum from the address inheritance QoS policy involves the compute node's virtualization management service obtaining complete information about the address inheritance QoS policy bound to the device by calling the API of the network service or storage service. The PCI address metadata is extracted from the policy's description field, and the checksum is extracted from the policy's rule field.

[0069] The steps for verifying PCI address metadata are as follows: First, the extracted PCI address metadata is processed using the same verification algorithm as during creation to calculate the current verification value. This current verification value is then compared with the verification value extracted from the rule field. If they match, the verification passes; otherwise, it fails. In the event of a verification failure, the system automatically reverts to the default dynamic address allocation logic, without affecting the normal startup of the virtual machine.

[0070] After successful verification, the steps for parsing the PCI address information involve first performing the reverse decoding and decompression operations on the PCI address metadata, as during creation, to restore the structured data. Core information such as the PCI address and bus type is then extracted from this structured data.

[0071] The steps for creating a device address configuration object are as follows: Create a corresponding address configuration object based on the bus type. For PCI bus devices, create a PCI address configuration object; for SCSI bus devices, create a SCSI address configuration object. Write the parsed PCI address information into the corresponding attribute of the address configuration object.

[0072] This step, which prewrites the device address configuration object into the device's configuration properties, overriding the dynamic address allocation logic, is executed before the virtualization management service generates the virtual machine configuration file. The prewritten address configuration object is directly used by the virtualization management service, and the dynamic address allocation logic is not executed again. The generated virtual machine configuration file contains the prewritten PCI address, which the device will use after the virtual machine starts.

[0073] In optional embodiments, the method, such as Figure 8 As shown, it also includes: S410: When the device is destroyed, the corresponding address inheritance QoS policy is automatically deleted; S420: When the PCI address of the device changes, the PCI address metadata and minimum bandwidth rules in the address inheritance QoS policy are updated synchronously.

[0074] In this embodiment, the step of automatically deleting the corresponding address-inherited QoS policy when a device is destroyed is implemented by listening to the device's destruction event. When the network service or storage service detects that the device has been destroyed, it automatically triggers the policy deletion process, calling the API to delete the corresponding address-inherited QoS policy. This step can prevent the accumulation of useless QoS policies and the occupation of database storage space.

[0075] When a device's PCI address changes, the process of synchronously updating the PCI address metadata and minimum bandwidth rules in the address inheritance QoS policy is implemented by monitoring the device's address change event. When the virtualization management service detects a change in the device's PCI address, it automatically triggers the policy update process, regenerating the PCI address metadata, updating the description and rule fields of the address inheritance QoS policy, and simultaneously updating the NUMA node and physical network interface card information in the minimum bandwidth rule. This step ensures that the metadata in the address inheritance QoS policy remains consistent with the actual state of the device.

[0076] In optional embodiments, the method, such as Figure 9 As shown, it also includes: S430: Create a shared address template QoS policy and write the PCI address templates of batch devices into the shared address template QoS policy; S440: Binds shared address template QoS policies to multiple devices; S450: Multiple devices read the PCI address template in the shared address template QoS policy and restore the PCI address of the corresponding serial number device.

[0077] In this embodiment, the step of creating a shared address template QoS policy is performed before batch deployment of virtual machines. A shared address template QoS policy refers to a QoS policy that can be bound to multiple devices simultaneously, and the policy includes PCI address templates for the batch of devices. The PCI address templates define fixed PCI addresses corresponding to devices with different serial numbers; for example, the first network interface card (NIC) corresponds to a first fixed address, the second NIC corresponds to a second fixed address, and the first data disk corresponds to a third fixed address.

[0078] The steps for binding a shared address template QoS policy to multiple devices are performed when creating a batch of virtual machines. All devices that need to use the same address template are bound to the same shared address template QoS policy.

[0079] The process of multiple devices reading the PCI address template from the shared address template QoS policy and restoring the PCI address of the corresponding serial number device is executed when each virtual machine starts. The virtualization management service reads the PCI address template from the shared address template QoS policy, matches the corresponding fixed PCI address according to the device serial number, and restores the address in the reserved slot. This step ensures that the PCI addresses of all virtual machines deployed in the same batch are completely consistent for devices with the same serial number, enabling zero-configuration deployment of batch services.

[0080] It's important to note that the key to implementing this method lies in the fact that mainstream cloud management platforms (such as OpenStack) limit their QoS policy APIs for network services (Neutron) and storage services (Cinder) to verifying the data type and value range of the field itself (e.g., DSCP values ​​are integers from 0 to 63). They don't perform in-depth verification of whether the rule is technically bound to and effective with a real network traffic marker or disk I / O rate limiting operation. Similarly, the description field is a plain text field with no formatting restrictions. Therefore, this method cleverly exploits the platform's lack of semantic verification. By taking the modulo of the verification value (e.g., taking the prefix of the SHA256 hash modulo 64) and filling it into the DSCP field, it fully complies with the platform's API verification rules. This successfully uses the QoS rule field as a reliable data storage slot that moves throughout the device's lifecycle. This process does not rely on any unpublished or specially authorized APIs; it is entirely based on the platform's publicly available, standard functionalities.

[0081] This application also provides a PCI device address persistence system in a virtualized environment, such as... Figure 10 As shown, it includes: The policy creation module 01 is used to create an address inheritance QoS policy for the device to be persisted, write the device's PCI address metadata into the address inheritance QoS policy, and add a minimum bandwidth rule to the address inheritance QoS policy. The minimum bandwidth rule is bound to the NUMA node and physical network card corresponding to the PCI address. The scheduling reservation module 02 is used to reserve slots for corresponding PCI addresses for devices according to the minimum bandwidth rule; the address recovery module is used to read the PCI address metadata in the address inheritance QoS policy and restore the PCI address of the device in the reserved slot to persist the PCI device address.

[0082] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for persistent PCI device address in a virtualized environment, characterized in that, include: Create an address inheritance QoS policy for the device to be persisted, write the device's PCI address metadata into the address inheritance QoS policy, and add a minimum bandwidth rule to the address inheritance QoS policy. The minimum bandwidth rule is bound to the NUMA node and physical network card corresponding to the PCI address. According to the minimum bandwidth rule, the scheduler reserves slots for corresponding PCI addresses for the devices; Read the PCI address metadata from the address inheritance QoS policy, restore the device's PCI address in the reserved slot, and persist the PCI device address.

2. The method according to claim 1, characterized in that, Create an address inheritance QoS policy for the device to be persisted, including: Obtain the device's PCI address information and device identifier; Encode the PCI address information and device identifier to generate PCI address metadata; Create an address inheritance QoS policy and write the PCI address metadata into the description field of the address inheritance QoS policy.

3. The method according to claim 2, characterized in that, The PCI address information and device identifier are encoded to generate PCI address metadata, including: Organize PCI address information and device identifiers into structured data; The structured data is compressed; the compressed structured data is then encoded to generate PCI address metadata. Calculate the checksum of the PCI address metadata and write the checksum into the rule field of the address inheritance QoS policy.

4. The method according to claim 1, characterized in that, Creating address inheritance QoS policies for devices to be persisted also includes: Check whether the detection device has been bound to the service QoS policy; If the device is already bound to a service QoS policy, create an address-inherited QoS sub-policy and set the parent policy of the address-inherited QoS sub-policy to the service QoS policy. Bind the address inheritance QoS sub-policy to the device.

5. The method according to claim 1, characterized in that, Create an address inheritance QoS policy for the device to be persisted, including: When the device is a network interface card (NIC) device, call the network service API to create a network-side address inheritance QoS policy; When the device is a disk device, call the storage service API to create a storage-side address that inherits the QoS policy.

6. The method according to claim 1, characterized in that, According to the minimum bandwidth rule, the scheduler reserves slots for corresponding PCI addresses for the devices, including: The scheduler identifies the address bound to the device and inherits the minimum bandwidth rule in the QoS policy; Filter out compute nodes that contain the corresponding physical network interface card and have available slots in the NUMA node; On the selected computing nodes, slots corresponding to PCI addresses are reserved for the devices, and the slots are marked as occupied.

7. The method according to claim 1, characterized in that, Read the PCI address metadata from the address inheritance QoS policy, restore the device's PCI address in the reserved slot, and persist the PCI device address, including: Read the PCI address metadata and checksum from the address inheritance QoS policy; The PCI address metadata is verified, and the PCI address information is obtained after the verification is successful. Create a device address configuration object and write the PCI address information into the device address configuration object; The device address configuration object is prewritten into the device's configuration properties, overriding the dynamic address allocation logic.

8. The method according to claim 1, characterized in that, Also includes: When a device is destroyed, the corresponding address inheriting the QoS policy is automatically deleted. When the PCI address of a device changes, the PCI address metadata and minimum bandwidth rules in the address inheritance QoS policy are updated synchronously.

9. The method according to claim 1, characterized in that, Also includes: Create a shared address template QoS policy, and write the PCI address templates of a batch of devices into the shared address template QoS policy; Bind a shared address template QoS policy to multiple devices; Multiple devices read the PCI address template from the shared address template QoS policy and restore the PCI address of the corresponding serial number device.

10. A PCI device address persistence system in a virtualized environment, characterized in that, include: The policy creation module is used to create an address inheritance QoS policy for the device to be persisted, write the device's PCI address metadata into the address inheritance QoS policy, and add a minimum bandwidth rule to the address inheritance QoS policy. The minimum bandwidth rule is bound to the NUMA node and physical network card corresponding to the PCI address. The scheduling reservation module is used to reserve slots for corresponding PCI addresses for devices according to the minimum bandwidth rule; The address recovery module is used to read the PCI address metadata in the address inheritance QoS policy, recover the PCI address of the device in the reserved slot, and persist the PCI device address.

Images