NVMe equipment operating system and method

Abstract

The invention provides an NVMe device operating system and method, and is applied to the technical field of virtualization, the system comprises a first NVMe device and a virtual machine, the virtual machine is provided with a first NVMe device driver, a user space of a host operating system is provided with a first NVMe device model, and the first NVMe device model is used for mapping a memory space of the first NVMe device; the virtual machine is used for realizing data direct connection between the virtual machine and the first NVMe equipment data surface through the first NVMe equipment driver and the first NVMe equipment model; after the virtual machine executes operation on the data in the first NVMe equipment in a data plane direct connection mode, the virtual machine is further used for notifying the first NVMe equipment through the control plane. According to the embodiment of the invention, the data plane direct connection between the virtual machine and the NVMe equipment is realized, the NVMe equipment is directly connected to the virtual machine, the physical performance of the NVMe equipment is fully exerted, and the data processing rate is improved.

Description

Technical Field

[0001] The present application relates to the field of virtualization technology, and in particular to a non-volatile memory host controller interface specification NVMe device operating system and method. Background Art

[0002] As a high-performance, highly scalable storage protocol, the non-volatile memory host controller interface specification (NVMHCIS, also known as NVM express, NVMe) is increasingly widely used, and various cloud vendors have provided high-performance storage devices based on NVMe. NVMe storage devices have the characteristics of fast data processing speed.

[0003] In the cloud scenario, the host uses virtual machine technology to create a virtual machine on the host operating system, and the client operating system runs on the virtual machine. Applications are installed on the virtual machine. When the user operates the application, the background instructions need to be passed from the user space of the virtual machine to the kernel space of the virtual machine, and then the kernel space of the virtual machine is trapped into the kernel space of the host operating system. After being processed in the kernel space of the host operating system, the data is set to the NVMe device; after the data is set to the NVMe device, the NVMe device replies to the virtual machine. The response first passes through the kernel space of the host operating system, then traps into the kernel space of the virtual machine, and finally traps out of the kernel space of the virtual machine and passes it to the front end of the application. The data instructions passed from the virtual machine to the NVMe device need to be constantly trapped and exited, and context switching is performed, resulting in slow data processing speed, which makes it impossible to play the fast data speed characteristics of the NVMe device. Summary of the invention

[0004] The present application provides a non-volatile memory host controller interface specification NVMe device operating system and method, the system realizes data plane direct connection between a virtual machine and an NVMe device, connects the NVMe device directly to the virtual machine, and gives full play to the physical performance of the NVMe device.

[0005] In a first aspect, the present application provides a non-volatile memory host controller interface specification NVMe device operating system, including: a processor and a first NVMe device, the processor is used to run a virtual machine on a host operating system through virtualization technology, and the processor runs in the user space of the host operating system or the kernel space of the host operating system;

[0006] A first NVMe device driver is provided in the virtual machine;

[0007] The user space of the host operating system is provided with a first NVMe device model, wherein the first NVMe device model is used to map the memory space of the first NVMe device;

[0008] Using the existing virtual host management system in the kernel space of the host operating system, a first data plane acceleration device and a first acceleration device driver module are provided in the virtual host management system;

[0009] A backend module of the first data plane acceleration device is provided in the user space of the host operating system;

[0010] A second data plane acceleration device and a second acceleration device driver module are provided in the kernel space of the host operating system;

[0011] The virtual machine is used to realize direct communication between the virtual machine and the data plane of the first NVMe device through the first NVMe device driver and the first NVMe device model;

[0012] After the virtual machine performs an operation on the data in the first NVMe device in a data plane direct passthrough manner, the virtual machine is further used to notify the operation on the data in the first NVMe device through the control plane, including:

[0013] The first NVMe device model is used to receive a first instruction sent by the virtual machine, where the first instruction is used to instruct the virtual machine to perform an operation on the data in the first NVMe device;

[0014] The first NVMe device model is also used to send the first instruction from the user space to the first data plane acceleration device in the virtual host management system through the back-end module, the first acceleration device driver module is used to drive the first data plane acceleration device to send the first instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the first instruction to the first NVMe device.

[0015] By setting the first NVMe device model in the user space of the host operating system, the first NVMe device model maps the memory space of the first NVMe device, so that the memory space of the first NVMe device is directly presented to the virtual machine, and the virtual machine can directly operate the first NVME device, thereby realizing direct data plane connection between the virtual machine and the first NVME device. After the virtual machine performs an operation on the data in the first NVME device, the first virtual machine notifies the first NVME device through the control plane path: the operation performed by the virtual machine on the first NVME device. In the traditional method, data instructions are repeatedly trapped and transmitted between the virtual machine front end, the virtual machine back end, the user space and the kernel space of the host operating system, which consumes a lot of time and makes it impossible to bring out the characteristics of the NVMe device. Compared with the traditional method, the present application operates on the data by means of direct data plane connection, which improves the data processing speed and efficiency, and can bring out the characteristics of the NVMe device. In addition, the bytes included in data instructions are usually longer, and the bytes included in control instructions are shorter. The time consumed by the traditional method of trapping and exiting the data instructions multiple times between the virtual machine operating system and the host operating system is longer than the time consumed by the control instructions on the control plane in the present application. Therefore, the method provided by the present application has faster data processing speed and higher efficiency.

[0016] Based on the first aspect, in a possible implementation, during startup of the virtual machine:

[0017] The first NVMe device model is further used to obtain a second instruction and send the second instruction to the back-end module, where the second instruction includes any one of an instruction for setting the number of queues, an instruction for setting an interrupt, and an instruction for setting a read-write mechanism;

[0018] The back-end module is used to send the second instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device, the first acceleration device driver module is used to drive the first data plane acceleration device to send the second instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the second instruction to the first NVMe device, thereby realizing the setting of the control plane.

[0019] Based on the first aspect, in a possible implementation, the first NVMe device is provided with an input-output memory management unit IOMMU, the IOMMU is used to manage the memory space of the first NVMe device, and the kernel space of the host operating system is provided with an IOMMU domain and an IOMMU driver;

[0020] During the startup of the virtual machine:

[0021] The first NVMe device model is further used to obtain a third instruction and send the third instruction to the backend module, wherein the third instruction is used to set a queue memory space, and the third instruction carries a virtual address of the queue memory space;

[0022] The backend module is used to send the third instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device;

[0023] The first data plane acceleration device is used to call the translation lookaside buffer TLB to map the virtual address of the queue memory space to a physical address, and pass the mapping relationship between the virtual address and the physical address to the IOMMU domain, and the IOMMU domain is used to mark the mapping relationship between the virtual address and the physical address as the first NVMe device;

[0024] The IOMMU domain is used to send the mapping relationship between the virtual address marked as the first NVMe device and the physical address to the IOMMU through the IOMMU driver, wherein the TLB includes the mapping relationship between the virtual address and the physical address of the first NVMe device; the IOMMU is used to save the mapping relationship between the virtual address and the physical address;

[0025] The first acceleration device driver module is used to drive the first data plane acceleration device to send the third instruction to the second data plane acceleration device; the second acceleration device driver module is used to drive the second data plane acceleration device to set the third instruction to the first NVMe device, thereby realizing the setting of the control plane.

[0026] Based on the first aspect, in a possible implementation, the second data plane acceleration device is provided with a plurality of data processing units DPU with different hardware characteristics and operation functions on the first NVMe device, and the second data plane acceleration device is used to implement adaptation of DPUs with different hardware characteristics;

[0027] The system is also used to implement hot migration of the virtual machine based on the first NVMe device model, the back-end module, the first data plane acceleration device and the second data plane acceleration device.

[0028] Since different DPUs have different operation methods for NVMe devices, during hot migration, if the DPU on the host where the virtual machine is located before migration is different from the DPU on the host where the virtual machine is located after migration, hot migration cannot be achieved. In order to solve this problem, in the present application solution, multiple operation functions for the first NVMe device are set in the second data plane acceleration device for DPUs with different hardware characteristics to support DPUs with different hardware characteristics, thereby achieving hot migration of virtual machines.

[0029] Based on the first aspect, in a possible implementation, the back-end module includes an NVMe protocol parsing submodule, an interrupt instruction processing submodule, and a queue instruction processing submodule.

[0030] Based on the first aspect, in a possible implementation, the first NVMe device driver is a native driver provided by the first NVMe device manufacturer.

[0031] Based on the first aspect, in a possible implementation,

[0032] When the data in the first NVMe device is operated by other devices, the other devices are not the virtual machine, and the first NVMe device is also used to report the updated data to the virtual machine through the data plane directly, and the first NVMe device is also used to notify the virtual machine through the control plane: the updated data has been reported.

[0033] Based on the first aspect, in a possible implementation, the virtual host management system includes a virtual vhost subsystem.

[0034] In a second aspect, the present application provides a non-volatile memory host controller interface specification NVMe device operation method, which is applied to a system, wherein the system includes a processor and a first NVMe device, the processor is used to run a virtual machine on a host operating system through virtualization technology, the processor runs in the user space of the host operating system or the kernel space of the host operating system, and a first NVMe device driver is provided in the virtual machine; a first NVMe device model is provided in the user space of the host operating system, wherein the first NVMe device model is used to map the memory space of the first NVMe device; utilizing the existing virtual host management system in the kernel space of the host operating system, a first data plane acceleration device and a first acceleration device driver module are provided in the virtual host management system; a back-end module of the first data plane acceleration device is provided in the user space of the host operating system; a second data plane acceleration device and a second acceleration device driver module are provided in the kernel space of the host operating system; the method includes:

[0035] The virtual machine is used to realize direct communication between the virtual machine and the data plane of the first NVMe device through the first NVMe device driver and the first NVMe device model;

[0036] After the virtual machine performs an operation on the data in the first NVMe device in a data plane direct passthrough manner, the virtual machine is further used to notify the operation on the data in the first NVMe device through the control plane, including:

[0037] The first NVMe device model is used to receive a first instruction sent by the virtual machine, where the first instruction is used to instruct the virtual machine to perform an operation on the data in the first NVMe device;

[0038] The first NVMe device model is also used to send the first instruction from the user space to the first data plane acceleration device in the virtual host management system through the back-end module, the first acceleration device driver module is used to drive the first data plane acceleration device to send the first instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the first instruction to the first NVME device.

[0039] Based on the second aspect, in a possible implementation, before the virtual machine is used to implement direct communication between the virtual machine and the first NVMe device data plane through the first NVMe device driver and the first NVMe device model, the method further includes:

[0040] During the startup of the virtual machine:

[0041] The first NVMe device model is further used to obtain a second instruction and send the second instruction to the back-end module, where the second instruction includes any one of an instruction for setting the number of queues, an instruction for setting an interrupt, and an instruction for setting a read-write mechanism;

[0042] The back-end module is used to send the second instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device, the first acceleration device driver module is used to drive the first data plane acceleration device to send the second instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the second instruction to the first NVMe device, thereby realizing the setting of the control plane.

[0043] Based on the second aspect, in a possible implementation, the first NVMe device is provided with an input-output memory management unit IOMMU, the IOMMU is used to manage the memory space of the first NVMe device, and the kernel space of the host operating system is provided with an IOMMU domain and an IOMMU driver;

[0044] The method further comprises: during the startup of the virtual machine:

[0045] The first NVMe device model is further used to obtain a third instruction and send the third instruction to the backend module, wherein the third instruction is used to set a queue memory space, and the third instruction carries a virtual address of the queue memory space;

[0046] The backend module is used to send the third instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device;

[0047] The first data plane acceleration device is used to call the translation lookaside buffer TLB to map the virtual address of the queue memory space to a physical address, and pass the mapping relationship between the virtual address and the physical address to the IOMMU domain, and the IOMMU domain is used to mark the mapping relationship between the virtual address and the physical address as the first NVMe device;

[0048] The IOMMU domain is used to send the mapping relationship between the virtual address marked as the first NVMe device and the physical address to the IOMMU through the IOMMU driver, wherein the TLB includes the mapping relationship between the virtual address and the physical address of the first NVMe device; the IOMMU is used to save the mapping relationship between the virtual address and the physical address;

[0049] The first acceleration device driver module is used to drive the first data plane acceleration device to send the third instruction to the second data plane acceleration device; the second acceleration device driver module is used to drive the second data plane acceleration device to set the third instruction to the first NVMe device, thereby realizing the setting of the control plane.

[0050] Based on the second aspect, in a possible implementation, the second data plane acceleration device is provided with a plurality of data processing units DPU with different hardware characteristics and operation functions on the first NVMe device, and the second data plane acceleration device is used to implement adaptation of DPUs with different hardware characteristics;

[0051] The method also includes: implementing hot migration of the virtual machine based on the first NVMe device model, the back-end module, the first data plane acceleration device, and the second data plane acceleration device.

[0052] Based on the second aspect, in a possible implementation, the first NVMe device driver is a native driver provided by the first NVMe device manufacturer.

[0053] Based on the second aspect, in a possible implementation,

[0054] When the data in the first NVMe device is operated by other devices, the other devices are not the virtual machine, and the first NVMe device is also used to report the updated data to the virtual machine through the data plane directly, and the first NVMe device is also used to notify the virtual machine through the control plane: the updated data has been reported.

[0055] Based on the second aspect, in a possible implementation, the virtual host management system includes a virtual vhost subsystem.

[0056] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each of the at least one computing device comprising a memory and a processor, the processor of the at least one computing device being used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the second aspect above and any possible implementation method of the second aspect.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, characterized in that it includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the above second aspect and any possible implementation method of the second aspect.

[0058] In a fifth aspect, the present application provides a computer program product comprising program instructions. When the program instructions are executed on a computing device cluster, the computing device cluster executes the method described in the second aspect or any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of a system structure provided for this application;

[0060] Figure 2 A schematic diagram of another system structure provided for this application;

[0061] Figure 3 A schematic diagram of the structure of a back-end module provided for this application;

[0062] Figure 4 A schematic diagram of another system structure provided for this application;

[0063] Figure 5 A schematic diagram of a computing device structure provided for this application;

[0064] Figure 6 A schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application are described below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and are not used to indicate or imply relative importance, nor are they used for specific reference. " / " represents an "or" relationship, such as A / B represents A or B.

[0066] A hypervisor is a simulator, which is software, firmware or hardware that creates or runs virtual machines. A host runs a hypervisor, which in turn runs one or more virtual machines. The hypervisor is used to allocate hardware resources such as processors, memory, and hard disks to one or more virtual machines, so that one or more virtual machines share the hardware resources on the host. The hypervisor is also used to create a virtual operating system for each virtual machine and manage the operation of each virtual operating system.

[0067] User space is a memory area that ordinary applications can access. When a process or thread runs in user space, instructions can only access user space. Applications in user space cannot access kernel space at will. If an application needs to access kernel space or data on the hard disk, it needs to create a new thread or process to call the operating system interface to complete the switch from user state to kernel state. When a process or thread runs in user space, it is called user state, and when a process or thread runs in kernel space, it is called kernel state.

[0068] Kernel space is an area that the operating system kernel can access. Kernel space is independent of ordinary applications, that is, ordinary applications cannot access kernel space. Kernel space is a protected space. When a process or thread runs in kernel space, instructions can access any space.

[0069] Direct memory access (DMA) is a mechanism that allows data to be transferred directly between peripheral devices and host memory without the involvement of the processor.

[0070] Live migration, also known as dynamic migration or real-time migration, is the saving / restoring of virtual machines. It usually saves the running status of the entire virtual machine completely and can quickly restore it to the original hardware platform or even a different hardware platform. After the restoration, the virtual machine still runs smoothly and the user will not notice any difference.

[0071] The present application embodiment provides a system, see Figure 1 , Figure 1 A schematic diagram of the structure of an NVMe device operating system provided in this application. The system includes a processor (not shown in the figure) and a first NVMe device, the processor is used to run a first virtual machine on a host operating system through a virtual manager, and a client operating system is run on the first virtual machine.

[0072] A first NVMe device driver is provided in the first virtual machine. The first NVMe device driver may be a native driver provided by the first NVMe device manufacturer, or a driver provided by the user himself.

[0073] An NVMe device model is set in the user space of the host operating system. The NVMe device model is used to simulate the hardware characteristics of a general NVMe device. The hardware characteristics of the general NVMe device include an interrupt handling method, a memory space of the NVMe device, etc. The NVMe device model maps the memory space of the first NVMe device so that the memory space of the first NVMe device is presented to the first virtual machine, so that the first virtual machine can operate the first NVMe device, wherein the operations include reading data, writing data, and deleting data.

[0074] Utilizing the existing virtual host management system in the kernel space of the host operating system, a first data plane acceleration device and a first acceleration device driver module are set in the virtual host management system. A backend module of the first data plane acceleration device is set in the user space of the host operating system. The virtual host management system provides an application programming interface (API) interface to the user space, so the backend module of the first data plane acceleration device can communicate with the first data plane acceleration device through the API interface. A second data plane acceleration device and a second acceleration device driver module are set in the kernel space of the host operating system (not in the virtual host management system). Optionally, the virtual host management system can be a vhost subsystem.

[0075] The virtual host management system is a management software that performs secondary settings on the server software and related software of the operating system under the operating system. It is a platform for operators to manage virtual machines, etc., to achieve the purpose of quick management of virtual machines. For example, the virtual host management system can be a vhost subsystem.

[0076] Among them, the first virtual machine and the NVMe device model are both connected to the first bus, and the two can communicate through the first bus. The first bus can be a virtual peripheral component interconnect standard (PCI) bus or a virtual extended industry standard architecture (EISA) bus, etc. The vhost subsystem and the second data plane acceleration device are both connected to the second bus, and the two can communicate through the second bus. The second bus can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus, etc.

[0077] based on Figure 1 The system simulates the first NVMe device through the NVMe device model, and the NVMe device model maps the memory space of the first NVMe device. Based on the NVMe device model and the first NVMe device driver in the first virtual machine, the first virtual machine and the first NVMe device achieve direct data plane communication, that is, the first virtual machine can directly operate the data in the first NVMe device through the direct data plane, and the operation includes writing data, reading data, and deleting data. After the first virtual machine operates the data in the first NVMe device through the direct data plane, the first virtual machine is also used to notify the first NVMe device through the control plane: the first virtual machine operates on the data in the first NVMe device.

[0078] Before the first virtual machine notifies the first NVMe device through the control plane, the control plane needs to be configured first. The following describes how to configure the control plane. Figure 2 As shown, Figure 2 A schematic diagram of another system structure provided for an embodiment of the present application.

[0079] Figure 2In the system, an input / output memory management unit (IOMMU) is also provided in the system. The IOMMU can be integrated in the first NVMe device or deployed as an independent component on the host. The IOMMU is used to manage the memory space of the first NVMe device. Specifically, when data needs to be written to the first NVMe device, the IOMMU is used to write the data to the first NVMe device. When the data in the first NVMe device needs to be read, the IOMMU is used to read the data from the first NVMe device. When the data in the first NVMe device needs to be deleted, the IOMMU is used to delete the data from the first NVMe device. The IOMMU domain and the IOMMU driver are set in the kernel space of the host operating system.

[0080] Configure the control plane, including configuring the queue's memory space, the number of queues, interrupts, and read-write mechanisms. The following first describes how to configure the number of queues, interrupts, and read-write mechanisms, and then describes how to configure the queue's memory space.

[0081] In one example, during the startup of the first virtual machine, the NVMe device model is used to obtain a second instruction, and the second instruction includes any one or more of an instruction for configuring / setting the number of queues, an instruction for configuring / setting interrupts, and an instruction for configuring / setting a read / write mechanism. The second instruction may be sent from the first virtual machine to the NVMe device model, or may be generated by the NVMe device model. For example, if the second instruction is used to set the read / write mechanism, the second instruction is sent from the first virtual machine to the NVMe device model, if the second instruction is used to set the number of queues, the second instruction is generated by the NVMe device model, and if the second instruction is used to set an interrupt, the second instruction is sent from the virtual machine manager (hypervisor) to the NVMe device model. The second instruction may also be other instructions, for example, the second instruction may also be an instruction for initialization, which is not limited in this application.

[0082] The NVMe device model is also used to send the second instruction to the back-end module of the first data plane acceleration device. The back-end module of the first data plane acceleration device is used to send the second instruction from the user space to the kernel space in the form of a character device through the API interface, specifically, to the first data plane acceleration device in the vhost subsystem of the kernel space. That is, the back-end module controls the first data plane acceleration device through the character device. When the second instruction is different, the character device used to control the first data plane acceleration device is different. For example, when the second instruction is used to set the number of queues, the character device vhost-ndpa-1 can be used to control the first data plane acceleration device. When the second instruction is used to set an interrupt, the character device vhost-ndpa-2 can be used to control the first data plane acceleration device. When the second instruction is used to set a read and write mechanism, the character device vhost-ndpa-3 can be used to control the first data plane acceleration device, and so on, where ndpa is the abbreviation of nvmedatapathacceleration.

[0083] Optionally, the backend module of the first data plane acceleration device may include an NVMe protocol parsing submodule, an interrupt instruction processing submodule, and a queue instruction processing submodule, for example, see Figure 3 The schematic diagram shown. The NVMe protocol parsing submodule is used to parse the protocol. When the second instruction is used to set the read-write mechanism, the NVMe protocol parsing submodule is used to parse the second instruction to determine the protocol supported by the control plane. The interrupt instruction processing submodule is used to process the instruction for setting the interrupt, and the queue instruction processing submodule is used to process the instruction for setting the number of queues. Therefore, the back-end module of the first data plane acceleration device has the ability to recognize and process instructions, and can determine the character device to be used according to the second instruction.

[0084] The first data plane acceleration device is used to send the second instruction from the vhost subsystem to the second data plane acceleration device under the drive of the first acceleration device driver module, that is, the second instruction exits from the vhost subsystem and reaches the kernel space of the main operating system. In other words, the first acceleration device driver module is used to drive the first data plane acceleration device to send the second instruction from the vhost subsystem to the second data plane acceleration device. Specifically, the first data plane acceleration device transfers the second instruction to the second bus, and the second data plane acceleration device obtains the second instruction from the second bus.

[0085] The second data plane acceleration device is used to set the second instruction to the first NVMe device under the drive of the second acceleration device driver module. In other words, the second acceleration device driver module is used to drive the second data plane acceleration device to set the second instruction to the first NVMe device. The second data plane acceleration device includes the host operating system's operation method for the first NVMe device. Specifically, the second data plane acceleration device is provided with a plurality of functions, and the second data plane acceleration device calls the corresponding function according to the second instruction, thereby realizing the setting of the first NVMe device.

[0086] The above examples can be used to set the number of queues, interrupts, read-write mechanisms, etc., so as to achieve the setting of the number of queues, interrupts, and read-write mechanisms in the control plane.

[0087] In another example, during the startup of the first virtual machine, the NVMe device model is used to obtain a third instruction, the third instruction is used to set / configure the memory space of the queue, and the third instruction carries the virtual address of the queue memory space. The NVMe device model is also used to send the third instruction to the back-end module of the first data plane acceleration device, and the back-end module is used to send the third instruction from the user space to the first data plane acceleration device in the vhost subsystem in the kernel space by means of a character device. Optionally, the queue instruction processing submodule in the back-end module is used to process the third instruction and determine the character device used. For example, when the third instruction is for setting the memory space of the queue, the back-end module can use the character device vhost-ndpa-4 to control the first data plane acceleration device.

[0088] The first data plane acceleration device is used to call the translation lookaside buffer (TLB) to map the virtual address of the queue memory space to a physical address. Wherein, the TLB is a functional module in the vhost subsystem, which stores the mapping relationship between the virtual address and the physical address in the first NVMe device. The first data plane acceleration device is also used to send the mapping relationship between the virtual address and the physical address of the queue memory space to the IOMMU domain, and the IOMMU domain is used to mark the mapping relationship, and mark the mapping relationship between the virtual address and the physical address of the queue memory space as corresponding to the first NVMe device. Wherein, the IOMMU domain corresponds to the first NVMe device. The vhost subsystem provides an API interface for IOMMU, and the first data plane acceleration device sends the mapping relationship to the IOMMU domain through the API interface for IOMMU. The IOMMU domain is used to send the mapping relationship between the virtual address and the physical address marked as the first NVMe device to the IOMMU through the IOMMU driver, and the IOMMU saves the mapping relationship between the virtual address and the physical address of the first NVMe device.

[0089] The first data plane acceleration device is also used to send the third instruction to the second data plane acceleration device under the drive of the first acceleration device driver module. In other words, the first acceleration device driver module is used to drive the first data plane acceleration device to send the third instruction to the second data plane acceleration device. The second data plane acceleration device is also used to set the third instruction to the first NVMe device under the drive of the second acceleration device driver module. In other words, the second acceleration device driver module is used to drive the second data plane acceleration device to set the third instruction to the first NVMe device, thereby realizing the setting of the queue memory space on the control plane.

[0090] After the control plane configuration / setting is completed, direct data communication between the first virtual machine and the first NVMe device can be achieved. Optionally, the first virtual machine can operate the data in the first NVMe device through direct memory access (DMA) technology, including writing data, reading data, and deleting data; when the data in the first NVMe device is operated by other devices, such as other devices writing data to the first NVMe device, where the other devices refer to non-first virtual machines, the first NVMe device can also send updated data to the first virtual machine through DMA technology.

[0091] When the first virtual machine performs an operation on the data in the first NVMe device, whether it is writing data, reading data, or deleting data, it needs to go through the IOMMU, and the IOMMU manages and operates the first NVMe device. Optionally, the IOMMU can also manage and operate the data in the first NVMe device through DMA technology.

[0092] After the first virtual machine performs an operation on the first NVMe device by means of data plane data direct pass-through, the first virtual machine is further used to notify the operation performed on the data in the first NVMe device through the control plane, specifically including:

[0093] The first NVMe device model is used to receive a first instruction sent by a first virtual machine, where the first instruction is used to indicate an operation performed by the first virtual machine on data in the first NVMe device. For example, the first instruction may be used to indicate that the first virtual machine has written data to the first NVMe device, or the first instruction may be used to indicate that the first virtual machine has read data from the first NVMe device, or the first instruction may be used to indicate that the first virtual machine has deleted data in the first NVMe device.

[0094] The first NVMe device model is also used to send the first instruction to the back-end module of the first data plane acceleration device, and the back-end module is used to parse the first instruction and determine the character device used to control the first data plane acceleration device according to the type of the first instruction. The type of the first instruction includes any one of the instructions for indicating that the first virtual machine has written data to the first NVMe device, read data from the first NVMe device, and deleted data in the first NVMe device. In the case of different types of the first instruction, the character device used by the back-end module is different. For example, when the first instruction is used to indicate that the first virtual machine has written data to the first NVMe device, the character device used by the back-end module may be vhost-ndpa-5, when the first instruction is used to indicate that the first virtual machine has read data from the first NVMe device, the character device used by the back-end module may be vhost-ndpa-6, and when the first instruction is used to indicate that the first virtual machine has deleted data in the first NVMe device, the character device used by the back-end module may be vhost-ndpa-7.

[0095] The back-end module of the first data plane acceleration device is also used to send the first instruction to the first data plane acceleration device in the vhost subsystem in the form of a character device through an API interface; the first data plane acceleration device is used to send the first instruction to the second data plane acceleration device under the drive of the first acceleration device driver module; the second data plane acceleration device is used to set the first instruction to the first NVMe device under the drive of the second acceleration device driver module to notify the first NVMe device of the operation performed by the first virtual machine.

[0096] In the case where the data in the first NVMe device is operated by other devices, and the other devices are not the first virtual machine, the first NVMe device is also used to report the updated data in the first NVMe device to the first virtual machine by means of data plane direct passthrough, and the first NVMe device is also used to notify the first virtual machine through the control plane: the updated data in the first NVMe device has been reported. For example, in the case where new data is written to the first NVMe device, the first NVMe device is used to send the written data to the first virtual machine by means of data plane direct passthrough, and is also used to notify the first virtual machine through the control plane: the newly written data in the first NVMe device has been reported.

[0097] In order to achieve hot migration of virtual machines, Figure 1 or Figure 2In the described system, the second data plane acceleration device is provided with operation functions for the first NVMe device for data processing units (DPUs) with multiple different hardware characteristics, and the second data plane acceleration device provides a unified interface for adapting different DPUs. For example, the second data plane acceleration device includes a DPU with a first hardware characteristic, a DPU with a second hardware characteristic, and a DPU with a third hardware characteristic, wherein the DPU with the first hardware characteristic, the DPU with the second hardware characteristic, and the DPU with the third hardware characteristic may be provided by the same manufacturer or by different manufacturers. The second data plane acceleration device is used to implement DPUs that adapt to different hardware characteristics. When hot migration of a virtual machine is performed, the DPUs on the two hosts before and after the migration may have different hardware characteristics. When the DPUs on the two hosts before and after the migration have different hardware characteristics, the second data plane acceleration device may select the corresponding operation function according to the hardware characteristics of the DPU on the host to implement the operation of the first NVMe device. For example, before migration, the first virtual machine is located on the first host, and the first host is deployed with a DPU with first hardware characteristics. In this case, when processing instructions, the second data plane acceleration device operates the first NVMe device according to the operation function corresponding to the first hardware characteristic DPU; after hot migration, the first virtual machine is used on the second host, and the second host is deployed with a DPU with second hardware characteristics. In this case, after migration, the second data plane acceleration device operates the first NVMe device according to the operation function corresponding to the second hardware characteristic DPU.

[0098] This application also provides a system architecture diagram, see Figure 4 , Figure 4 A schematic diagram of another system architecture provided for this application. Figure 4 In the schematic diagram, multiple virtual machines are virtualized on a host, the multiple virtual machines include a first virtual machine and a second virtual machine, and the host includes a first NVMe device, a second NVMe device, a third NVMe device and a fourth NVMe device, wherein the first NVMe device and the second NVMe device are used to serve the first virtual machine, and the third NVMe device and the fourth NVMe device are used to serve the second virtual machine.

[0099] A first NVMe device model is provided in the user space of the host operating system, and the first NVMe device model is used to simulate the general characteristics of the first NVMe device and maps the memory space of the first NVMe device; a second NVMe device model is also provided, and the second NVMe device model is used to simulate the general characteristics of the second NVMe device and maps the memory space of the second NVMe device. Similarly, a third NVMe device and a fourth NVMe device are provided in the user space of the host operating system, and the third NVMe device model is used to simulate the general characteristics of the third NVMe device and maps the memory space of the third NVMe device, and the fourth NVMe device model is used to simulate the general characteristics of the fourth NVMe device and maps the memory space of the fourth NVMe device.

[0100] By using the existing virtual vhost subsystem in the kernel space of the host operating system, data plane acceleration device 1, data plane acceleration device 3, data plane acceleration device 5, data plane acceleration device 7 and a first acceleration device driver module are respectively set in the vhost subsystem. Data plane acceleration device 2, data plane acceleration device 4, data plane acceleration device 6, data plane acceleration device 8 and a second acceleration device driver module are set in the kernel space of the host operating system (outside the vhost subsystem). The back-end module of the data plane acceleration device 1, the back-end module of the data plane acceleration device 3, the back-end module of the data plane acceleration device 5 and the back-end module of the data plane acceleration device 7 are respectively set in the user space of the host operating system.

[0101] The host operating system kernel space also has an IOMMU domain and an IOMMU driver. The IOMMU can be deployed on an independent device.

[0102] The functions of the above devices and modules are Figure 1 , Figure 2 The functions of the corresponding device modules are similar. For details, please refer to Figure 1 and Figure 2 The description of the corresponding content is not repeated here for the sake of brevity of the specification. Based on the above system, the first virtual machine can directly communicate with the first NVMe device and the second NVMe device, and the second virtual machine can directly communicate with the third NVMe device and the fourth NVMe device.

[0103] On the control plane, the first NVMe device model, the backend module of the data plane acceleration device 1, the data plane acceleration device 1, the first acceleration device driver module, the data plane acceleration device 2, and the second acceleration device driver module constitute a control plane path, and the first virtual machine can send control plane instructions to the first NVMe device through the control plane path. Similarly, the second NVMe device model, the backend module of the data plane acceleration device 3, the data plane acceleration device 3, the first acceleration device driver module, the data plane acceleration device 4, and the second acceleration device driver module constitute a control plane path, and the first virtual machine can send control plane instructions to the second NVMe device through the control plane path. The third NVMe device model, the backend module of the data plane acceleration device 5, the data plane acceleration device 5, the first acceleration device driver module, the data plane acceleration device 6, and the second acceleration device driver module constitute a control plane path, and the second virtual machine can send control plane instructions to the third NVMe device through the control plane path. The fourth NVMe device model, the back-end module of the data plane acceleration device 7, the data plane acceleration device 7, the first acceleration device driver module, the data plane acceleration device 8, and the second acceleration device driver module constitute a control plane path, and the second virtual machine can send control plane instructions to the fourth NVMe device through the control plane path.

[0104] Before implementing data plane data pass-through, the system needs to be configured. During configuration, the IOMMU domain is used to mark which NVMe device the mapping relationship between the virtual address and the physical address in the instruction belongs to, and the marked mapping relationship is sent to the IOMMU through the IOMMU driver so that the IOMMU can save the mapping relationship.

[0105] The present application provides a computing device 500, see Figure 5 , Figure 5 A schematic diagram of the structure of a computing device 500 provided in this application. The computing device 500 can be configured as Figure 1 or Figure 2 The NVMe device operating system, the computing device may be a physical host, a server, etc., and the computing device 500 includes:

[0106] Bus 502, processor 504, memory 506, and communication interface 508. Processor 504, memory 506, and communication interface 508 communicate with each other via bus 502. It should be understood that the present application does not limit the number of processors and memories in computing device 500.

[0107] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus 502 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 502 may include a path for transmitting information between various components of the computing device 500 (eg, the memory 506, the processor 504, and the communication interface 508).

[0108] The processor 504 may include any one or more of a central processing unit (CPU), a DPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0109] The memory 506 may include a volatile memory, such as a random access memory (RAM). The processor 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0110] The memory 506 stores executable code, and the processor 504 executes the executable code to implement an NVMe device operation method. That is, the memory 506 stores instructions for executing an NVMe device operation method.

[0111] The communication interface 508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0112] The present application provides a computing device cluster, which includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center.

[0113] like Figure 6As shown, the computing device cluster includes at least one computing device 500. The memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing an NVMe device operation method. In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc.

[0114] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes an NVMe device operation method.

[0115] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute a scheduling method.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-volatile memory host controller interface specification NVMe device operating system, characterized in that: include: A processor and a first NVMe device, wherein the processor is used to run a virtual machine on a host operating system through virtualization technology, and the processor runs in a user space of the host operating system or a kernel space of the host operating system; A first NVMe device driver is provided in the virtual machine; The user space of the host operating system is provided with a first NVMe device model, wherein the first NVMe device model is used to map the memory space of the first NVMe device; Using the existing virtual host management system in the kernel space of the host operating system, a first data plane acceleration device and a first acceleration device driver module are provided in the virtual host management system; A backend module of the first data plane acceleration device is provided in the user space of the host operating system; A second data plane acceleration device and a second acceleration device driver module are provided in the kernel space of the host operating system; The virtual machine is used to realize direct communication between the virtual machine and the data plane of the first NVMe device through the first NVMe device driver and the first NVMe device model; After the virtual machine performs an operation on the data in the first NVMe device in a data plane direct passthrough manner, the virtual machine is further used to notify the operation on the data in the first NVMe device through the control plane, including: The first NVMe device model is used to receive a first instruction sent by the virtual machine, where the first instruction is used to instruct the virtual machine to perform an operation on the data in the first NVMe device; The first NVMe device model is also used to send the first instruction from the user space to the first data plane acceleration device in the virtual host management system through the back-end module, the first acceleration device driver module is used to drive the first data plane acceleration device to send the first instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the first instruction to the first NVMe device.

2. The system according to claim 1, characterized in that During the startup of the virtual machine: The first NVMe device model is further used to obtain a second instruction and send the second instruction to the back-end module, where the second instruction includes any one or more of an instruction for setting the number of queues, an instruction for setting an interrupt, and an instruction for setting a read-write mechanism; The back-end module is used to send the second instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device, the first acceleration device driver module is used to drive the first data plane acceleration device to send the second instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the second instruction to the first NVMe device, thereby realizing the setting of the control plane.

3. The system according to claim 1 or 2, characterized in that: An input-output memory management unit (IOMMU) is provided on the first NVMe device, and the IOMMU is used to manage the memory space of the first NVMe device. An IOMMU domain and an IOMMU driver are provided in the kernel space of the host operating system; During the startup of the virtual machine: The first NVMe device model is further used to obtain a third instruction and send the third instruction to the backend module, wherein the third instruction is used to set a queue memory space, and the third instruction carries a virtual address of the queue memory space; The backend module is used to send the third instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device; The first data plane acceleration device is used to call the translation lookaside buffer TLB to map the virtual address of the queue memory space to a physical address, and pass the mapping relationship between the virtual address and the physical address to the IOMMU domain, and the IOMMU domain is used to mark that the mapping relationship between the virtual address and the physical address corresponds to the first NVMe device; The IOMMU domain is used to send the mapping relationship between the virtual address marked as the first NVMe device and the physical address to the IOMMU through the IOMMU driver, wherein the TLB includes the mapping relationship between the virtual address and the physical address of the first NVMe device; the IOMMU is used to save the mapping relationship between the virtual address and the physical address; The first acceleration device driver module is used to drive the first data plane acceleration device to send the third instruction to the second data plane acceleration device; the second acceleration device driver module is used to drive the second data plane acceleration device to set the third instruction to the first NVMe device, thereby realizing the setting of the control plane.

4. The system according to any one of claims 1 to 3, characterized in that: The second data plane acceleration device is provided with a plurality of data processing units DPU with different hardware characteristics and operation functions on the first NVMe device, and the second data plane acceleration device is used to realize adaptation of DPUs with different hardware characteristics; The system is also used to implement hot migration of the virtual machine based on the first NVMe device model, the back-end module, the first data plane acceleration device and the second data plane acceleration device.

5. The system according to any one of claims 1 to 4, characterized in that: The back-end module includes an NVMe protocol parsing submodule, an interrupt instruction processing submodule and a queue instruction processing submodule.

6. The system according to any one of claims 1 to 5, characterized in that: The first NVMe device driver is a native driver provided by the first NVMe device manufacturer.

7. The system according to any one of claims 1 to 6, characterized in that: When the data in the first NVMe device is operated by other devices, the other devices are not the virtual machine, and the first NVMe device is also used to report the updated data to the virtual machine through the data plane directly, and the first NVMe device is also used to notify the virtual machine through the control plane: the updated data has been reported.

8. A non-volatile memory host controller interface specification NVMe device operation method, characterized in that: The invention is applied to a system, wherein the system includes a processor and a first NVMe device, wherein the processor is used to run a virtual machine on a host operating system through virtualization technology, wherein the processor runs in the user space of the host operating system or the kernel space of the host operating system, and wherein a first NVMe device driver is provided in the virtual machine; wherein a first NVMe device model is provided in the user space of the host operating system, wherein the first NVMe device model is used to map the memory space of the first NVMe device; wherein a first data plane acceleration device and a first acceleration device driver module are provided in the virtual host management system by utilizing an existing virtual host management system in the kernel space of the host operating system; wherein a backend module of the first data plane acceleration device is provided in the user space of the host operating system; wherein a second data plane acceleration device and a second acceleration device driver module are provided in the kernel space of the host operating system; and wherein the method includes: The virtual machine is used to realize direct communication between the virtual machine and the data plane of the first NVMe device through the first NVMe device driver and the first NVMe device model; After the virtual machine performs an operation on the data in the first NVMe device in a data plane direct passthrough manner, the virtual machine is further used to notify the operation on the data in the first NVMe device through the control plane, including: The first NVMe device model is used to receive a first instruction sent by the virtual machine, where the first instruction is used to instruct the virtual machine to perform an operation on the data in the first NVMe device; The first NVMe device model is also used to send the first instruction from the user space to the first data plane acceleration device in the virtual host management system through the back-end module, the first acceleration device driver module is used to drive the first data plane acceleration device to send the first instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the first instruction to the first NVME device.

9. The method according to claim 8, characterized in that Before the virtual machine is used to implement direct communication between the virtual machine and the first NVMe device data plane data through the first NVMe device driver and the first NVMe device model, the method further includes: During the startup of the virtual machine: The first NVMe device model is further used to obtain a second instruction and send the second instruction to the back-end module, where the second instruction includes any one of an instruction for setting the number of queues, an instruction for setting an interrupt, and an instruction for setting a read-write mechanism; The back-end module is used to send the second instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device, the first acceleration device driver module is used to drive the first data plane acceleration device to send the second instruction from the virtual host management system to the second data plane acceleration device, and the second acceleration device driver module is used to drive the second data plane acceleration device to set the second instruction to the first NVMe device, thereby realizing the setting of the control plane.

10. The method according to claim 8 or 9, characterized in that: The first NVMe device is provided with an input-output memory management unit IOMMU, and the IOMMU is used to manage the memory space of the first NVMe device, and the kernel space of the host operating system is provided with an IOMMU domain and an IOMMU driver; The method further comprises: during the startup of the virtual machine: The first NVMe device model is further used to obtain a third instruction and send the third instruction to the backend module, wherein the third instruction is used to set a queue memory space, and the third instruction carries a virtual address of the queue memory space; The backend module is used to send the third instruction from the user space to the first data plane acceleration device in the virtual host management system by means of a character device; The first data plane acceleration device is used to call the translation lookaside buffer TLB to map the virtual address of the queue memory space to a physical address, and pass the mapping relationship between the virtual address and the physical address to the IOMMU domain, and the IOMMU domain is used to mark the mapping relationship between the virtual address and the physical address as the first NVMe device; The IOMMU domain is used to send the mapping relationship between the virtual address marked as the first NVMe device and the physical address to the IOMMU through the IOMMU driver, wherein the TLB includes the mapping relationship between the virtual address and the physical address of the first NVMe device; the IOMMU is used to save the mapping relationship between the virtual address and the physical address; The first acceleration device driver module is used to drive the first data plane acceleration device to send the third instruction to the second data plane acceleration device; the second acceleration device driver module is used to drive the second data plane acceleration device to set the third instruction to the first NVMe device, thereby realizing the setting of the control plane.

11. The method according to any one of claims 8 to 10, characterized in that: The second data plane acceleration device is provided with a plurality of data processing units DPU with different hardware characteristics and operation functions on the first NVMe device, and the second data plane acceleration device is used to realize adaptation of DPUs with different hardware characteristics; The method also includes: implementing hot migration of the virtual machine based on the first NVMe device model, the back-end module, the first data plane acceleration device, and the second data plane acceleration device.

12. The method according to any one of claims 8 to 11, characterized in that The first NVMe device driver is a native driver provided by the first NVMe device manufacturer.

13. The method according to any one of claims 8 to 12, characterized in that: When the data in the first NVMe device is operated by other devices, the other devices are not the virtual machine, and the first NVMe device is also used to report the updated data to the virtual machine through the data plane directly, and the first NVMe device is also used to notify the virtual machine through the control plane: the updated data has been reported.

14. A computing device cluster, characterized in that: It includes at least one computing device, each of the at least one computing device includes a memory and a processor, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 8 to 13.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 8 to 13.

16. A computer program product, characterized in that The method comprises program instructions. When the program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 8 to 13.

Images