Virtual PCI device for multi-domain isolated communication and implementation method thereof

Abstract

The application discloses a virtual PCI device for multi-domain isolated communication and an implementation method thereof, which comprises the following steps: dividing the memory mapping IO address space of the virtual PCI device into a general configuration space and a private configuration space, storing a first type of register for generally configuring a plurality of data channels in the general configuration space, and storing a second type of register for specially configuring a single data channel in the private configuration space. The protocol limitation of the virtual PCI device can be broken, and a plurality of data channels can be served by one virtual PCI device.

Description

Technical Field

[0001] This application relates to the field of virtual PCI device technology, and in particular to a virtual PCI device for multi-domain isolated communication and its implementation method. Background Technology

[0002] In current paravirtualization solutions, data transfer between the virtual machine and the host environment relies on PCI virtual devices implemented with QEMU. These devices use specific MMIO ports (memory-mapped I / O address spaces) to negotiate shared memory addresses with the virtual machine. In multi-domain isolation applications, the required functionality for each partition has the following characteristics: 1. Dynamic addition and reduction: During system operation, whether it's new user access or new user data sharing and transmission, multiple new communication channels need to be established. Therefore, the number of communication channels must be able to dynamically adjust. 2. Varied content and format: Different communication channels handle different interaction tasks, requiring different initialization processes at both ends and transmitting different data formats. Therefore, communication channels should be adaptable to different application scenarios and support a sufficiently rich set of configuration and data types.

[0003] Traditional virtual PCI devices are limited by the PCI bus, which restricts the number of devices. The ability to add or remove devices at runtime is greatly affected by the kernel and platform.

[0004] In existing virtual machine data path implementations, shared memory is generally used to accelerate the data exchange process between the virtual machine and the host machine. The construction method of shared memory is as follows:

[0005] 1. The virtual machine allocates a contiguous block of physical memory;

[0006] 2. The virtual machine writes the memory address and length to the configuration space of the virtual PCI device;

[0007] 3. In QEMU, the virtual PCI device handler converts the obtained memory address and length into an address that the host machine can directly process;

[0008] 4. If data transfer is required later, the virtual machine only needs to write the data to the physical memory allocated in the virtual machine and then notify the virtual PCI device.

[0009] In the traditional implementation described above, a virtual PCI device can only have a segment of shared memory and serve one data path. If the virtual machine needs multiple data paths, it can only be accomplished by simulating multiple virtual PCI devices. Dynamically increasing or decreasing the number of data paths at runtime also depends on the hot-plugging of virtual PCI devices, which is quite limiting. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention proposes a virtual PCI device and its implementation method for multi-domain isolated communication, which can overcome the protocol limitations of virtual PCI devices and serve multiple data paths through a single virtual PCI device.

[0011] This invention proposes a method for implementing a virtual PCI device for multi-domain isolated communication, comprising:

[0012] The memory-mapped I / O address space of the virtual PCI device is divided into a general configuration space and a private configuration space. The general configuration space stores a first type of register for general configuration of multiple data paths, and the private configuration space stores a second type of register for dedicated configuration of a single data path.

[0013] Optionally, it also includes:

[0014] Set the number of private configuration spaces, where each private configuration space has a second type of register for dedicated configuration of a single data path;

[0015] Each data path is assigned a unique number. The first type of register performs general configuration on the currently active data path. Then, based on the number assigned to the currently active data path, it switches to the second type of register in the private configuration space to perform dedicated configuration on the currently active data path.

[0016] Optionally, the first type of register performs general configuration for the currently active data path, including:

[0017] Provides the partition with read / write access to the basic attributes of the currently active data path;

[0018] The partition is provided with the number of the data path currently experiencing an interruption. The partition learns the reason for the interruption by querying other fields related to the interruption and completes the interruption handling by resetting the interruption status.

[0019] Optionally, the first type of register includes:

[0020] cur_index is used to configure the currently active data path number;

[0021] tunnel_count is used to configure the number of private configuration spaces;

[0022] tunnel_state is used to configure the construction status of the currently active data path;

[0023] tunnel_error is used to configure the error status of the currently active data path;

[0024] irq_index is used to configure the number of the data path that is currently interrupted;

[0025] irq_status is used to configure the reason for the current interruption;

[0026] `tunnel_function` is used to configure the type number of the currently active data path;

[0027] tunnel_consume is used to notify the currently active data path of new events.

[0028] Optionally, the second type of register includes:

[0029] Avail_ring_size is used to configure the size of Avail_ring;

[0030] Used_ring_size is used to configure the size of the Used_ring;

[0031] Desc_size is used to configure the size of Ring_desc_table;

[0032] Addr is used to configure the starting address of the shared memory;

[0033] Len is used to configure the length of shared memory.

[0034] In another aspect, the present invention proposes a virtual PCI device for multi-domain isolated communication. The memory-mapped I / O address space of the virtual PCI device is divided into a general configuration space and a private configuration space. The general configuration space stores a first type of register for general configuration of multiple data paths, and the private configuration space stores a second type of register for dedicated configuration of a single data path.

[0035] Optionally, the first type of register includes:

[0036] cur_index is used to configure the currently active data path number;

[0037] tunnel_count is used to configure the number of private configuration spaces;

[0038] tunnel_state is used to configure the construction status of the currently active data path;

[0039] tunnel_error is used to configure the error status of the currently active data path;

[0040] irq_index is used to configure the number of the data path that is currently interrupted;

[0041] irq_status is used to configure the reason for the current interruption;

[0042] `tunnel_function` is used to configure the type number of the currently active data path;

[0043] tunnel_consume is used to notify the currently active data path of new events.

[0044] Optionally, the second type of register includes:

[0045] Avail_ring_size is used to configure the size of Avail_ring;

[0046] Used_ring_size is used to configure the size of the Used_ring;

[0047] Desc_size is used to configure the size of Ring_desc_table;

[0048] Addr is used to configure the starting address of the shared memory;

[0049] Len is used to configure the length of shared memory.

[0050] Optionally, both the first type of register and the second type of register have a width of 32 bits.

[0051] As can be seen from the above technical solutions, the technical solutions of the present invention have the following advantages:

[0052] 1. In previous implementations, to support multiple data paths, it was necessary to create a corresponding number of virtual PCI devices in the virtual machine; however, in the design of this patent, multiple data path creation and processing can be achieved by setting multiple private configuration spaces in one virtual PCI device. No matter how many data paths there are, only one virtual PCI device is needed, which simplifies the information of the virtual PCI device seen by the virtual machine.

[0053] 2. The data path is exposed to the virtual machine entirely through the virtual PCI device, so the register contents of the virtual PCI device can be adjusted at runtime to dynamically adjust the number of data paths;

[0054] 3. The number of data paths is exposed to the virtual machine through the tunnel_count register in the general configuration space. Therefore, its upper limit is determined by the size of the tunnel_count register, which can break through the protocol limitations of virtual PCI devices and implement multiple data paths with a small performance cost. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the MMIO design provided for an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the initialization process of a virtual machine system provided in an embodiment of the present invention;

[0058] Figure 3 A schematic diagram illustrating the construction of a data path provided in an embodiment of the present invention;

[0059] Figure 4 This is a processor system architecture based on the PCI bus. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0061] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0062] An embodiment of the present invention provides a method for implementing a virtual PCI device for multi-domain isolated communication, comprising:

[0063] The memory-mapped I / O address space of the virtual PCI device is divided into a general configuration space and a private configuration space. The general configuration space stores a first type of register for general configuration of multiple data paths, and the private configuration space stores a second type of register for dedicated configuration of a single data path.

[0064] The virtual PCI device is a PCI device virtualized by QEMU, see [link / reference]. Figure 1This embodiment achieves the design goal of a virtual PCI device serving multiple data paths by dividing the MMIO (memory-mapped I / O address space) of the PCI device into two parts: general and private.

[0065] To ensure unified management and forwarding of all fields, the register width in all configuration spaces is forcibly set to 32 bits to avoid parsing complexity caused by inconsistent lengths and widths. The settings for the first type of registers in the general configuration space are shown in the table below.

[0066]

[0067] The working process of the virtual PCI device universal configuration space includes:

[0068] The virtual PCI device assigns a unique number to each data path and uses `tunnel_count` to provide the partition with the number of data paths. This design allows the partition to switch the currently configured and used data path by reading and writing the `cur_index` field in the general configuration space. The currently used data path will be referred to as the active path. The general configuration space provides the partition with read / write access to some basic attributes of the active data path, such as its function type and current working status. To support data paths in sending processing requests to the partition, the virtual PCI device sends requests to the partition via interrupts. Therefore, the virtual PCI device provides interrupt-related fields in parallel and provides the partition with the number of the data path currently experiencing an interrupt via `irq_index`. The partition can query other interrupt-related fields to learn the specific reason for the interrupt and complete the interrupt handling by resetting the interrupt status.

[0069] It should be noted that the general configuration space is the configuration space content shared by multiple data paths, and changes to the register content in the general configuration space will affect all data paths; while the private configuration space is private to each data path, so changes to the content of a private configuration space will only affect the data path to which it belongs.

[0070] The following example illustrates how a private configuration space can be implemented.

[0071] In a simple multi-domain isolation system, the following three data paths need to be integrated within the partition:

[0072] 1. A data path used to obtain and set partition information and related configurations for multi-domain isolation systems;

[0073] 2. A data path used for transferring files between partitions;

[0074] 3. A data path used to transmit partitioning behavior to the host machine for dynamic verification;

[0075] All three data paths described above can achieve data transmission using a circular queue built on shared memory. A circular queue consists of the following three structures:

[0076] 1. ring_desc_table: It maintains the available ring_desc structures in shared memory by using an array of ring_desc structures as a pool of ring_desc structures;

[0077] 2. avail_ring: A circular queue where partitions act as write ends and QEMU acts as read ends. Each element is a number in the ring_desc_table, pointing to the ring_desc structure being used. This structure is used by partitions to pass the ring_desc containing the data to QEMU.

[0078] 3. `used_ring`: A circular queue where QEMU acts as the write end and the partition as the read end. Each element is a `ring_desc_table` number and a related processing structure feedback value. The number points to the `ring_desc` number passed from `avail_ring`, used to pass the `ring_desc` that has been processed by QEMU.

[0079] Under this structure, each data path requires the following type II registers in its private configuration space:

[0080]

[0081]

[0082] In the scenarios of the three data paths described above, the contents of the private configuration space for each data path are shown in the table above, and the private configuration spaces for each data path are independent of each other. In the general configuration space, tunnel_count = 3.

[0083] See Figure 2 The initialization process of the virtual machine system using the virtual PCI device implementation method of this invention is as follows:

[0084] exist Figure 2 In this patent, pguest_mm is the virtual PCI device driver corresponding to the configuration space, partition_mm is the simulation program corresponding to the virtual PCI device, and pguest_tunnel_state is the state description structure created by the virtual machine for each data path. The content of this structure is not within the scope of this patent.

[0085] After the virtual machine initializes, the virtual PCI device driver first reads the contents of the `tunnel_count` register and then loops `tunnel_count` times. In each loop, a `pguest_tunnel_state` structure is created and initialized. The `cur_index` in the general configuration space is set to the current loop number, used to switch the virtual PCI device's private configuration space to the corresponding data path configuration space. Then, the contents of this private configuration space (in this case, the contents of the `avail_ring_size`, `used_ring_size`, and `desc_size` registers) are read. Based on these register contents, the size of the shared memory is calculated, and contiguous physical memory is allocated. Finally, by writing to the `addr` and `len` registers in the private configuration space, the physical memory is transferred to the virtual PCI device's simulation program, `partition-mm`, ​​completing the creation of the data path.

[0086] When data needs to be transferred, the virtual machine only needs to write the data to the corresponding shared physical memory, then write to cur_index to switch the private configuration space, and finally write to the tunnel_consume register to notify the corresponding data path that there is new data.

[0087] Once the data processing is complete, the virtual PCI device emulation program reports to the virtual machine which data path has finished processing by modifying the contents of the irq_index and irq_status registers.

[0088] This invention also proposes a virtual PCI device for multi-domain isolated communication. The memory-mapped I / O address space of the virtual PCI device is divided into a general configuration space and a private configuration space. The general configuration space stores a first type of register for general configuration of multiple data paths, and the private configuration space stores a second type of register for dedicated configuration of a single data path.

[0089] Furthermore, the first type of register includes:

[0090] cur_index is used to configure the currently active data path number;

[0091] tunnel_count is used to configure the number of private configuration spaces;

[0092] tunnel_state is used to configure the construction status of the currently active data path;

[0093] tunnel_error is used to configure the error status of the currently active data path;

[0094] irq_index is used to configure the number of the data path that is currently interrupted;

[0095] irq_status is used to configure the reason for the current interruption;

[0096] `tunnel_function` is used to configure the type number of the currently active data path;

[0097] tunnel_consume is used to notify the currently active data path of new events.

[0098] Furthermore, the second type of register includes:

[0099] Avail_ring_size is used to configure the size of Avail_ring;

[0100] Used_ring_size is used to configure the size of the Used_ring;

[0101] Desc_size is used to configure the size of Ring_desc_table;

[0102] Addr is used to configure the starting address of the shared memory;

[0103] Len is used to configure the length of shared memory.

[0104] Furthermore, both the first type of register and the second type of register have a width of 32 bits.

[0105] The embodiments of the present invention have the following advantages:

[0106] 1. In the prior art, to support multiple data paths, a corresponding number of virtual PCI devices must be created in the virtual machine; however, in the design of this embodiment, no matter how many data paths there are, only one virtual PCI device is needed, which simplifies the information of the virtual PCI device seen by the virtual machine.

[0107] 2. The data path is fully exposed to the virtual machine through the virtual PCI device, so the register contents of the virtual PCI device can be adjusted at runtime to dynamically adjust the number of data paths.

[0108] 3. The number of data paths is exposed to the virtual machine through the tunnel_count register in the general configuration space. Therefore, its upper limit is determined by the size of the tunnel_count register. In this patent, the size of tunnel_count is limited to 32 bits, so the number of data paths is limited to 2^32.

[0109] The technical terms and related technical explanations used in this invention are as follows:

[0110] Data path: A data path is a message path for data transfer constructed to complete a specific task within a virtual machine. In this invention, the construction of the data path relies on shared memory. See also Figure 3 A data path requires the collaborative construction of partitions, QEMU, and the host kernel. Within a partition, `pguest_tunnelN` is the front-end processor, responsible for producing and consuming data within the partition. `partition_tunnelN` is a memory translator, translating the memory address containing the data passed from the partition into a memory address that the host machine can directly process. `phost_tunnelN` is the data processing program in the host kernel, handling data forwarding and verification. These three components are connected via shared memory. This patent specifically addresses how QEMU manages the translation services for these data paths (`partition_tunnelN`) by modifying the virtual PCI device configuration space, and how it presents the data path's status information to the partition through the configuration space.

[0111] Partitioning: In a multi-domain isolation architecture, a partition is physically a virtual machine simulated using QEMU. However, unlike a regular virtual machine, a partition integrates multi-domain isolation related service modules, such as data path processing modules.

[0112] PCI Protocol: The PCI bus is a peripheral component interconnection standard proposed by Intel in 1992, and the PCI Special Interest Group (PCI-SIG) was established to maintain the bus protocol. The introduction of PCI devices solved three problems: speed, scalability, and configurability. As an extension bus for the CPU, the initial version of the PCI bus provided a bandwidth of 133MB / s, far exceeding the performance requirements of computers at the time. Furthermore, the PCI bus is connected to the CPU through a host bridge device and can be expanded through a tree structure, using PCI bridge devices to connect another PCI bus, thus greatly increasing the number of PCI devices supported in the system. The PCI bus defines the configuration space format for each PCI device, and the operating system can configure PCI devices by reading and writing to the PCI configuration space. The processor system architecture based on the PCI bus is as follows: Figure 4 As shown.

[0113] MMIO: Memory-mapped I / O (input / output) address space, is the memory address space pointed to by BaseAddress0 in the PCI configuration space structure.

[0114] QEMU: QEMU is a set of emulated processors written by Fabrice Bellard and distributed under the GPL license. It is widely used on the GNU / Linux platform.

[0115] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit one to the specific order or hierarchy described.

[0116] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that, on the contrary, the invention is presented with fewer features than all of the disclosed individual embodiments. Therefore, the appended claims are hereby clearly incorporated into the detailed description, with each claim representing an individual preferred embodiment of the invention.

[0117] It is impossible to describe all possible combinations of components or methods in order to describe the above embodiments; however, those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims.

Claims

1. A method for implementing a virtual PCI device for multi-domain isolated communication, characterized in that, include: The memory-mapped I / O address space of the virtual PCI device is divided into a general configuration space and a private configuration space. The general configuration space stores a first type of register for general configuration of multiple data paths, and the private configuration space stores a second type of register for dedicated configuration of a single data path. When initializing a virtual machine system using a virtual PCI device implementation method, the following steps are included: After the virtual machine is initialized, the virtual PCI device driver first reads the contents of the tunnel_count register, then loops through tunnel_count times. In each loop, it creates a pguest_tunnel_state structure, initializes its contents, and sets the cur_index in the general configuration space to the current loop number. This is used to switch the virtual PCI device's private configuration space to the corresponding data path configuration space. Then, it reads the register contents of the corresponding private configuration space, calculates the size of the shared memory based on these register contents, allocates contiguous physical memory, and then transfers the physical memory to the virtual PCI device's simulation program partition-mm by writing to the addr and len registers of the private configuration space, thus completing the creation of the data path. When data needs to be transmitted, the data is written to the corresponding shared physical memory, then written to cur_index to switch the private configuration space, and finally written to the tunnel_consume register to notify the corresponding data path that there is new data. Once the data processing is complete, the virtual PCI device emulation program reports to the virtual machine that the corresponding data path has finished processing by modifying the contents of the irq_index and irq_status registers.

2. The method of claim 1, wherein the method further comprises: Also includes: Set the number of private configuration spaces, where each private configuration space has a second type of register for dedicated configuration of a single data path; Each data path is assigned a unique number. The first type of register performs general configuration on the currently active data path. Then, based on the number assigned to the currently active data path, it switches to the second type of register in the private configuration space to perform dedicated configuration on the currently active data path.

3. The method of claim 2, wherein the method further comprises: The first type of register performs general configuration for the currently active data path, including: Provides the partition with read / write access to the basic attributes of the currently active data path; The partition is provided with the number of the data path currently experiencing an interruption. The partition learns the reason for the interruption by querying other fields related to the interruption and completes the interruption handling by resetting the interruption status.

4. The method of claim 2, wherein the method further comprises: The first type of register includes: cur_index is used to configure the currently active data path number; tunnel_count is used to configure the number of private configuration spaces; tunnel_state is used to configure the construction status of the currently active data path; tunnel_error is used to configure the error status of the currently active data path; irq_index is used to configure the number of the data path that is currently interrupted; irq_status, for configuring the reason of current interrupt; tunnel_function, for configuring the type number of current active data tunnel; tunnel_consume, for notifying that there is new event in current active data tunnel.

5. The method of claim 2, wherein the method further comprises: The second type of registers include: Avail_ring_size, for configuring the size of Avail_ring; Used_ring_size, for configuring the size of Used_ring; Desc_size, for configuring the size of Ring_desc_table; Addr, for configuring the first address of shared memory; Len, for configuring the length of shared memory.

6. A virtual PCI device for multi-domain isolated communication, comprising: The memory-mapped IO address space of the virtual PCI device is divided into a general configuration space and a private configuration space, the first type of registers for configuring multiple data tunnels are stored in the general configuration space, and the second type of registers for configuring a single data tunnel are stored in the private configuration space; The method for implementing the virtual PCI device comprises the following steps: After the virtual machine is initialized, the virtual PCI device driver first reads the content of the tunnel_count register, then loops tunnel_count times, creates a pguest_tunnel_state structure and initializes the content of the structure in each loop, sets the cur_index in the general configuration space as the current loop number to switch the private configuration space of the virtual PCI device to the configuration space of the corresponding data tunnel, then reads the register content of the corresponding private configuration space, calculates the size of the shared memory according to the register content and allocates continuous physical memory, and then transfers the physical memory to the simulation program partition-mm of the virtual PCI device by writing the addr and len registers in the private configuration space, to complete the creation of the data tunnel; When data needs to be transferred, the data is written to the corresponding shared memory physical memory, then the cur_index is written to switch the private configuration space, and finally the tunnel_consume register is written to notify the corresponding data tunnel of new data; When the data processing is completed, the virtual PCI device simulation program reports to the virtual machine that the corresponding data tunnel has completed the data processing by modifying the content of the irq_index and irq_status registers.

7. The virtual PCI device for multi-domain isolated communication according to claim 6, wherein, The first type of registers include: cur_index, for configuring the number of the current active data tunnel; tunnel_count, for configuring the number of private configuration spaces; tunnel_state, for configuring the construction state of the current active data tunnel; tunnel_error, for configuring the error state of the current active data tunnel; irq_index, for configuring the number of the data tunnel of current interrupt; irq_status, for configuring the reason of current interrupt; tunnel_function, used to configure the type number of the current active data tunnel; tunnel_consume, used to notify that there is a new event for the current active data tunnel.

8. The virtual PCI device for multi-domain isolated communication according to claim 6, wherein, The second type of registers includes: Avail_ring_size, used to configure the size of Avail_ring; Used_ring_size, used to configure the size of Used_ring; Desc_size, used to configure the size of Ring_desc_table; Addr, used to configure the first address of the shared memory; Len, used to configure the length of the shared memory.

9. The virtual PCI device for multi-domain isolated communication according to claim 6, wherein, The width of the first type of registers and the second type of registers is 32 bits.

Images