A deployment method, device and equipment of a virtual machine Ceph cluster and a medium

Abstract

The application discloses a virtual machine Ceph cluster deployment method and device, equipment and medium, and relates to the field of computer network communication, which comprises the following steps: running OVS-DPDK in the system on a chip of a target intelligent network card, creating a virtual bridge, creating corresponding ports for each physical function and each virtual function of the target intelligent network card displayed to the target host, and adding the ports to the virtual bridge; on the host side, modifying each target physical function and each target virtual function assigned to the virtual machine from the kernel driver to the user state driver, modifying the target non-volatile fast memory in the target host assigned to the virtual machine from the kernel driver to the user state driver; starting each virtual machine based on the bus device function identification number corresponding to each target physical function, each target virtual function and each target non-volatile fast memory, and deploying each virtual machine into a Ceph cluster by using a preset cluster deployment tool. The storage access performance of the virtual machine Ceph cluster is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer network communication, and particularly relates to a deployment method and device of a virtual machine Ceph cluster, equipment and medium. BACKGROUND

[0002] Ceph is a decentralized distributed storage system, which provides better performance, reliability and scalability. According to the scene division, Ceph can be divided into three blocks, which are object storage, block device and file system service. As a distributed storage system, a Ceph cluster needs multiple hosts for deploying Ceph-Mon (Monitor), Ceph-Mgr (Manager), Ceph-OSD (Object Storage Device) and other services. For application developers, a virtual machine cluster is often used for deployment, including Vmware, VirtualBox, KVM (Kernel-based Virtual Machine) and other virtual machine platforms.

[0003] The network connection between each virtual machine and the host is very important. For the Vmware platform, the network can be configured through the bridge mode or the NAT (Network Address Translation) mode, and each virtual machine and the host can be located in the same network segment to achieve interconnection. On the Linux KVM, there is also a similar network configuration method, such as configuring the bridge mode, and the communication between each node in the Ceph cluster is completed by relying on the soft switching of the host operating system.

[0004] The storage medium used for deploying Ceph in the virtual machine is generally a local file created on the host using the qemu-img command, and then it is transferred to the virtual machine as an IDE (Integrated Drive Electronics) / SCSI (Small Computer System Interface) device. In the virtual machine, each disk is added to the Ceph cluster as an OSD.

[0005] Current network forwarding relies on the host system kernel, and its performance is limited by the host's CPU (Central Processing Unit) capabilities and workload, especially when the host is running multiple virtual machines, such as in a Ceph virtual machine cluster with anywhere from three to over a dozen virtual machines. Under high CPU load, network forwarding performance may be limited, thus affecting the overall performance of the distributed storage system. Furthermore, accessing the actual physical storage medium from within the virtual machines also requires the host operating system; however, multiple kernel accesses can easily lead to performance degradation. Moreover, as virtual disks are file-based, multiple files may reside on the same physical disk; when multiple virtual machines access them concurrently, contention can occur, increasing response time and thus reducing the Ceph cluster's access performance.

[0006] In summary, how to deploy virtual machine Ceph clusters to improve the storage access performance of Ceph clusters is a problem that needs to be solved. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for deploying a Ceph virtual machine cluster, which can improve the storage access performance of the Ceph cluster. The specific solution is as follows:

[0008] Firstly, this application discloses a method for deploying a virtual machine Ceph cluster, including:

[0009] OVS-DPDK is run on the system-on-a-chip of the target smart network card, and a virtual bridge is created. Corresponding ports are also created for each physical function and each virtual function that the target smart network card exposes to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card that is pre-installed in the target host.

[0010] On the target host side, the target physical functions and target virtual functions assigned to the virtual machine are modified from kernel drivers to user-mode drivers, and the target non-volatile fast memory in the target host assigned to the virtual machine is modified from kernel drivers to user-mode drivers.

[0011] Each virtual machine is started based on the bus device function identifier corresponding to each of the target physical functions, target virtual functions, and target non-volatile fast memory, and each virtual machine is deployed as a Ceph cluster using a preset cluster deployment tool.

[0012] Optionally, before running OVS-DPDK on the target smart network card's on-chip system, the method further includes:

[0013] The physical server to be used to deploy the Ceph cluster is determined, and the physical server is designated as the target host; wherein the physical server is equipped with a predetermined number of non-volatile fast memories;

[0014] The target smart network interface card (NIC) is installed into the PCIe slot of the target host, and the target host is powered on. The target smart NIC includes a field-programmable gate array (FPGA) and a system-on-a-chip (SoC). After power-on, the target host uses the lspci command to view the physical functions displayed by the FPGA. After power-on, the SoC uses the lspci command to view the physical functions displayed by the FPGA.

[0015] Optionally, the field-programmable gate array (FPGA) uses a packet forwarding method based on a fast forwarding path for packet forwarding, while the system-on-a-chip (SoC) uses a packet forwarding method based on a slow forwarding path for packet forwarding.

[0016] Accordingly, the method further includes:

[0017] Pre-configure preset message forwarding rules applicable to the target communication process; the target communication process includes communication processes between different physical functions, communication processes between different virtual functions, communication processes between physical functions and virtual functions, and communication processes between physical functions and external hosts;

[0018] The preset message forwarding rules include:

[0019] If it is the first network communication packet, the first network communication packet is uploaded to the system on chip, so that the system on chip can create a corresponding flow table entry for the first network communication packet based on the slow forwarding path, and send the flow table entry and the first network communication packet to the field programmable gate array, so that the field programmable gate array can forward the first network communication packet based on the flow table entry and using the fast forwarding path;

[0020] If it is not the first network communication packet, the non-first network communication packet is uploaded to the field-programmable gate array (FPGA) so that the FPGA can forward the non-first network communication packet based on the flow table entry and using the fast forwarding path.

[0021] Optionally, the target smart network interface card includes a number of physical functions, and each physical function obtains multiple corresponding virtual functions through single root input / output virtualization technology.

[0022] Optionally, the modification of the target physical functions and target virtual functions assigned to the virtual machine from kernel drivers to user-mode drivers includes:

[0023] Call the dpdk-devbind.py tool provided by DPDK;

[0024] Determine the target physical functions and target virtual functions assigned to the virtual machine;

[0025] The dpdk-devbind.py tool is used to uninstall the virtio-pci drivers for each of the target physical functions and each of the target virtual functions, and to mount the vfio-pci drivers, so as to change each of the target physical functions and each of the target virtual functions from kernel drivers to user-space drivers.

[0026] Optionally, the step of changing the target non-volatile fast memory in the target host assigned to the virtual machine from a kernel driver to a user-mode driver includes:

[0027] Determine the target non-volatile fast memory assigned to the virtual machine from the target host;

[0028] The virtio-pci driver of the target non-volatile fast memory is unloaded using a virtual file system, and the vfio-pci driver is mounted to change the target non-volatile fast memory from a kernel driver to a user-mode driver.

[0029] Optionally, the deployment method for the virtual machine Ceph cluster further includes:

[0030] Within each virtual machine, network interfaces corresponding to each target physical function, each target virtual function, and each target non-volatile fast memory are determined respectively;

[0031] Configure each of the aforementioned network interfaces with an Internet Protocol address belonging to the same network segment.

[0032] Secondly, this application discloses a deployment apparatus for a virtual machine Ceph cluster, comprising:

[0033] A port addition module is used to run OVS-DPDK on the on-chip system of the target smart network card, create a virtual bridge, and create corresponding ports for each physical function and each virtual function that the target smart network card exposes to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card pre-installed in the target host;

[0034] The driver modification module is used to modify the target physical functions and target virtual functions assigned to the virtual machine from kernel drivers to user-mode drivers on the target host side, and to modify the target non-volatile fast memory in the target host assigned to the virtual machine from kernel drivers to user-mode drivers.

[0035] The cluster deployment module is used to start each virtual machine based on the bus device function identifier corresponding to each of the target physical functions, each of the target virtual functions, and each of the target non-volatile fast memory, and to deploy each of the virtual machines as a Ceph cluster using a preset cluster deployment tool.

[0036] Thirdly, this application discloses an electronic device, including:

[0037] Memory, used to store computer programs;

[0038] A processor for executing the computer program to implement the steps of the aforementioned disclosed method for deploying a virtual machine Ceph cluster.

[0039] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed method for deploying a virtual machine Ceph cluster.

[0040] As can be seen, this application runs OVS-DPDK on the system-on-a-chip of the target smart network card, creates a virtual bridge, and creates corresponding ports for each physical function and each virtual function of the target smart network card to be presented to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card pre-installed in the target host; on the target host side, each target physical function and each target virtual function assigned to the virtual machine is modified from kernel driver to user-mode driver, and the target non-volatile fast memory in the target host assigned to the virtual machine is modified from kernel driver to user-mode driver; each virtual machine is started based on the bus device function identifier corresponding to each target physical function, each target virtual function, and each target non-volatile fast memory, and each virtual machine is deployed as a Ceph cluster using a preset cluster deployment tool.

[0041] The beneficial effects of this application are as follows: The target smart network interface card (NIC) is pre-installed on the target host. By running OVS-DPDK on the NIC's on-chip system, creating a virtual bridge, and creating corresponding ports for each physical and virtual function of the target smart NIC exposed to the target host, all ports are added to the virtual bridge. This enables communication between physical functions, virtual functions, and physical and virtual functions on the target host side. Furthermore, on the target host side, the target physical and virtual functions assigned to the virtual machine are modified from kernel drivers to user-mode drivers, allowing the devices to be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system and directly control the NIC hardware, eliminating the overhead of repeated kernel switching. In other words, this application introduces smart NIC network hardware acceleration to improve network forwarding performance, increase network bandwidth, reduce network latency, and ultimately improve Ceph cluster storage access performance. Furthermore, on the target host side, the target non-volatile fast memory in the target host assigned to the virtual machine needs to be modified from a kernel driver to a user-mode driver so that the non-volatile fast memory device can be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system to directly read and write to the storage disk, reducing the host CPU load, reducing access time, and eliminating the CPU consumption and access time latency caused by kernel context switching, thereby improving the overall performance of the entire virtual machine Ceph cluster storage. Finally, this application starts each virtual machine based on the bus device function identifiers corresponding to each target physical function, each target virtual function, and each target non-volatile fast memory, and uses a preset cluster deployment tool to deploy each virtual machine into a high-performance Ceph cluster. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart of a method for deploying a virtual machine Ceph cluster disclosed in this application;

[0044] Figure 2 This is a schematic diagram of a specific Ceph cluster disclosed in this application;

[0045] Figure 3 This application discloses a flowchart of a specific method for deploying a Ceph virtual machine cluster.

[0046] Figure 4This is a schematic diagram of a Ceph cluster storage medium disclosed in this application;

[0047] Figure 5 This is a schematic diagram of the deployment device structure for a virtual machine Ceph cluster disclosed in this application;

[0048] Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0050] Current network forwarding relies on the host system kernel, and its performance is limited by the host CPU's capabilities and operating load, especially when the host is running multiple virtual machines, such as in a Ceph virtual machine cluster with anywhere from three to over a dozen virtual machines. Under high CPU load, network forwarding performance may be limited, thus affecting the overall performance of the distributed storage system. Furthermore, accessing the actual physical storage medium from within the virtual machines also requires the host operating system; however, multiple kernel accesses can easily lead to performance degradation. Moreover, as virtual disks are file-based, multiple files may reside on the same physical disk, triggering contention when multiple virtual machines access them concurrently, increasing response time, and thus reducing Ceph cluster access performance. Therefore, this application discloses a method, apparatus, device, and medium for deploying a virtual machine Ceph cluster, which can improve the storage access performance of the Ceph cluster.

[0051] See Figure 1 As shown in the figure, this application discloses a method for deploying a virtual machine Ceph cluster, the method including:

[0052] Step S11: Run OVS-DPDK on the system-on-a-chip of the target smart network card, create a virtual bridge, and create corresponding ports for each physical function and each virtual function that the target smart network card exposes to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card that is pre-installed in the target host.

[0053] In this embodiment, the target smart network interface card (NIC) has been pre-installed on the target host. It is understood that the smart NIC can be used to offload network, storage, and computing tasks. Through various hardware acceleration methods, it can effectively improve network bandwidth, reduce latency, and enhance system performance. The smart NIC can offload the load running on the server CPU, thereby saving server resources and improving overall efficiency.

[0054] This embodiment runs OVS-DPDK on the System-on-Chip (SOC) of the target smart network interface card (NIC), creates a virtual bridge, and creates corresponding ports for each Physical Function (PF) and Virtual Function (VF) that the target smart NIC exposes to the target host. By adding all ports to the virtual bridge, communication between physical functions, between virtual functions, and between physical and virtual functions can be achieved on the target host side. OVS-DPDK, or Open vSwitch with DPDK, is a high-performance virtual switch that combines DPDK (Data Plane Development Kit) and Open vSwitch (OVS) technologies to accelerate data plane processing in a virtualized environment. DPDK includes device drivers that map the PF / VF on the target host side to OVS ports by creating representation ports.

[0055] It should be noted that the target smart network card presents the target host with multiple virtio-net (one virtual Ethernet card) type PFs, and each PF can be virtualized into multiple VFs through SR-IOV (Single Root I / O Virtualization).

[0056] Step S12: On the target host side, the target physical functions and target virtual functions assigned to the virtual machine are modified from kernel drivers to user-mode drivers, and the target non-volatile fast memory in the target host assigned to the virtual machine is modified from kernel drivers to user-mode drivers.

[0057] In this embodiment, on the target host side, the target physical functions and target virtual functions assigned to the virtual machine are modified from kernel drivers to user-mode drivers so that the devices can be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system and directly control the network card hardware, eliminating the overhead caused by repeated kernel switching. That is, this application introduces intelligent network card network hardware acceleration to improve network forwarding performance, increase network bandwidth, reduce network latency, and achieve the goal of improving Ceph cluster storage access performance.

[0058] Furthermore, on the target host side, the target non-volatile memory express (NVMe) device in the target host designated for the virtual machine needs to be modified from the kernel driver to the user-mode driver so that the non-volatile memory device can be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system to directly read and write the storage disk, reducing the host CPU load, reducing access time, and eliminating the CPU consumption and access time latency caused by kernel context switching, thereby improving the overall performance of the entire virtual machine Ceph cluster storage.

[0059] Step S13: Start each virtual machine based on the bus device function identifier corresponding to each of the target physical functions, each of the target virtual functions, and each of the target non-volatile fast memories, and deploy each virtual machine as a Ceph cluster using a preset cluster deployment tool.

[0060] In this embodiment, the bus device function identifier (Bus, Device, Function, i.e., BDF) assigned to each target physical function and each target virtual function, as well as the bus device function identifier assigned to each target non-volatile fast memory, are used as device parameters to start the KVM virtual machine.

[0061] Furthermore, the above method also includes: determining the network interfaces corresponding to each of the target physical functions, target virtual functions, and target non-volatile fast memory within each virtual machine; and configuring Internet Protocol addresses (IP addresses) of the same network segment for each of the network interfaces. That is, in this embodiment, the network interfaces corresponding to each PF / VF are added to different netns within each virtual machine and configured with Internet Protocol addresses (IP addresses) of the same network segment to verify connectivity and bandwidth performance. Additionally, IP addresses of the same network segment are also configured for the network interfaces corresponding to each NVMe. The hosts file is edited to specify a corresponding hostname for each IP address, and the Ceph cluster is subsequently deployed based on these hostnames.

[0062] This embodiment pre-deploys a Ceph cluster using a pre-defined cluster deployment tool. Specifically, it uses ceph-deploy to deploy a Ceph cluster. First, the ceph-deploy deployment tool is installed, then the cluster is initialized, and Mon nodes, Mgr nodes, and OSD nodes are deployed sequentially. It should be noted that a Ceph system typically includes the following components: Monitor (Ceph-Mon), used to maintain the state of the cluster map; clients use this component for authentication when connecting to the Ceph cluster; OSD (Ceph-OSD), used for storing all data and objects in the cluster, with each disk corresponding to one OSD process; Manager (Ceph-Mgr), used to collect Ceph cluster status and operating metrics, providing a Ceph dashboard and resetful API; and MDS (Ceph-Mds), the metadata server, which provides metadata computation, caching, and synchronization services for the Ceph file system. Ceph's main advantage is its distributed storage. When storing each piece of data, the location of the data storage is calculated to ensure a balanced data distribution, eliminating the traditional single point of failure problem and allowing for horizontal scaling.

[0063] For example, such as Figure 2 As shown, a three-node Ceph cluster is deployed using VM0, VM1, and VM2. VM0 serves as Ceph-Mon, Ceph-Mgr, and Ceph-OSD simultaneously, while VM1 and VM2 serve as Ceph-OSD. The specific process is as follows:

[0064] ceph-deploy new VM0

[0065] ceph-deploymoncreate-initial

[0066] ceph-deploy mgr create VM0

[0067] ceph-deploy admin VM0

[0068] ceph-deploy admin VM1

[0069] ceph-deploy admin VM2

[0070] ceph-deploy disk zap VM0 / dev / nvme0n1

[0071] ceph-deploy disk zap VM0 / dev / nvme1n1

[0072] ceph-deploy disk zap VM0 / dev / nvme2n1

[0073] ceph-deploy osd create VM0--data / dev / nvme0n1

[0074] ceph-deploy osd create VM0--data / dev / nvme1n1

[0075] ceph-deploy osd create VM0--data / dev / nvme2n1

[0076] ceph-deploy disk zap VM1 / dev / nvme0n1

[0077] ceph-deploy disk zap VM1 / dev / nvme1n1

[0078] ceph-deploy disk zap VM1 / dev / nvme2n1

[0079] ceph-deploy osd create VM1--data / dev / nvme0n1

[0080] ceph-deploy osd create VM1--data / dev / nvme1n1

[0081] ceph-deploy osd create VM1--data / dev / nvme2n1

[0082] ceph-deploy disk zap VM2 / dev / nvme0n1

[0083] ceph-deploy disk zap VM2 / dev / nvme1n1

[0084] ceph-deploy disk zap VM2 / dev / nvme2n1

[0085] ceph-deploy osd create VM2--data / dev / nvme0n1

[0086] ceph-deploy osd create VM2--data / dev / nvme1n1

[0087] ceph-deploy osd create VM2--data / dev / nvme2n1.

[0088] Furthermore, this application embodiment can also configure block storage applications in a Ceph cluster, specifically by creating a replica storage pool, enabling RBD (Rados Block Device), initializing RBD, and creating an image. The specific process is as follows:

[0089] ceph osd pool create rbd-pool 512 512

[0090] ceph osd pool application enable rbd-pool rbd

[0091] rbd pool init-p rbd-pool

[0092] rbd create img0--size 1G--pool rbd-pool--image-feature layering

[0093] rbd create img1--size 1G--pool rbd-pool--image-feature layering.

[0094] In this embodiment, SPDK runs on the host machine. The application accesses the Ceph cluster by calling the librbd and librados libraries, creates a bdev using the rbd image in the cluster, and tests the Ceph cluster performance. Compiling SPDK requires installing the dependency packages librados-devel and librbd-devel and specifying the configure parameter --with-rbd. The runtime environment requires the libraries librados and librbd, as well as the configuration files ceph.conf and ceph.client.admin.keyring for accessing the Ceph cluster.

[0095] rpc.py bdev_rbd_create-b ceph0 rbd-pool img0 512.

[0096] In addition, remote hosts, acting as Ceph clients, access the Ceph cluster via a physical network connection established through a smart NIC. For example, if a remote host maps img1 to its local machine and executes `rbd map img1`, the storage device ` / dev / rbd0` can be seen using `lsblk`.

[0097] As can be seen, this application runs OVS-DPDK on the system-on-a-chip of the target smart network card, creates a virtual bridge, and creates corresponding ports for each physical function and each virtual function of the target smart network card to be presented to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card pre-installed in the target host; on the target host side, each target physical function and each target virtual function assigned to the virtual machine is modified from kernel driver to user-mode driver, and the target non-volatile fast memory in the target host assigned to the virtual machine is modified from kernel driver to user-mode driver; each virtual machine is started based on the bus device function identifier corresponding to each target physical function, each target virtual function, and each target non-volatile fast memory, and each virtual machine is deployed as a Ceph cluster using a preset cluster deployment tool.

[0098] The beneficial effects of this application are as follows: The target smart network interface card (NIC) is pre-installed on the target host. By running OVS-DPDK on the NIC's on-chip system, creating a virtual bridge, and creating corresponding ports for each physical and virtual function of the target smart NIC exposed to the target host, all ports are added to the virtual bridge. This enables communication between physical functions, virtual functions, and physical and virtual functions on the target host side. Furthermore, on the target host side, the target physical and virtual functions assigned to the virtual machine are modified from kernel drivers to user-mode drivers, allowing the devices to be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system and directly control the NIC hardware, eliminating the overhead of repeated kernel switching. In other words, this application introduces smart NIC network hardware acceleration to improve network forwarding performance, increase network bandwidth, reduce network latency, and ultimately improve Ceph cluster storage access performance. Furthermore, on the target host side, the target non-volatile fast memory in the target host assigned to the virtual machine needs to be modified from a kernel driver to a user-mode driver so that the non-volatile fast memory device can be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system to directly read and write to the storage disk, reducing the host CPU load, reducing access time, and eliminating the CPU consumption and access time latency caused by kernel context switching, thereby improving the overall performance of the entire virtual machine Ceph cluster storage. Finally, this application starts each virtual machine based on the bus device function identifiers corresponding to each target physical function, each target virtual function, and each target non-volatile fast memory, and uses a preset cluster deployment tool to deploy each virtual machine into a high-performance Ceph cluster.

[0099] See Figure 3As shown, this application embodiment discloses a specific method for deploying a virtual machine Ceph cluster. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0100] Specifically, it includes:

[0101] Step S21: Determine the physical server for deploying the Ceph cluster and designate the physical server as the target host; wherein the physical server is equipped with a preset number of non-volatile fast memories.

[0102] In this embodiment, the equipment required for deploying the Ceph cluster is first selected. Specifically, a high-performance physical server is selected as the target host, and the physical server is equipped with multiple NVMe disks (e.g., 16) as storage media. Figure 4 This is a schematic diagram of a Ceph cluster storage medium provided in an embodiment of this application.

[0103] Step S22: Install the target smart network card into the PCIe slot of the target host and power on the target host; wherein, the target smart network card includes a field-programmable gate array and a system-on-a-chip.

[0104] In this embodiment, a target smart network interface card (NIC) is provided for network offloading. The target smart NIC is installed in the server's PCIe slot and then powered on. The target smart NIC in this embodiment consists of an FPGA (Field-Programmable Gate Array) and a SoC (System on Chip). The SoC software provides flexible functionality, while the FPGA handles hardware acceleration logic; the combination of the two achieves both flexibility and high efficiency.

[0105] Understandably, after power-on, the target host can use the `lspci` command to view the various physical functions of the virtio-net type displayed by the field-programmable gate array (FPGA); similarly, after power-on, the system-on-a-chip (SoC) can use the `lspci` command to view the various physical functions of the virtio-net type displayed by the FPGA. `lspci` is a tool used to display all PCI bus devices in the system or all devices connected to that bus.

[0106] Step S23: Run OVS-DPDK on the system-on-a-chip of the target smart network card, create a virtual bridge, and create corresponding ports for each physical function and each virtual function that the target smart network card exposes to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card that is pre-installed in the target host.

[0107] In this embodiment, communication between each PF / VF can be achieved through a virtual bridge.

[0108] It should be noted that the field-programmable gate array (FPGA) uses a packet forwarding method based on a fast forwarding path for packet forwarding, while the system-on-a-chip (SoC) uses a packet forwarding method based on a slow forwarding path for packet forwarding and issues hardware forwarding table entries.

[0109] Accordingly, the above method further includes: pre-configuring preset message forwarding rules applicable to the target communication process; the target communication process includes communication processes between different physical functions, communication processes between different virtual functions, communication processes between physical functions and virtual functions, and communication processes between physical functions and external hosts; wherein, the preset message forwarding rules include: if it is the first network communication message, then the first network communication message is uploaded to the system-on-a-chip, so that the system-on-a-chip creates a corresponding flow table entry for the first network communication message based on a slow forwarding path, and sends the flow table entry and the first network communication message to the field-programmable gate array, so that the field-programmable gate array forwards the first network communication message based on the flow table entry and using the fast forwarding path; if it is not the first network communication message, then the non-first network communication message is uploaded to the field-programmable gate array, so that the field-programmable gate array forwards the non-first network communication message based on the flow table entry and using the fast forwarding path. In other words, in this application, the first network communication packet is first sent to the SoC for processing. After the slow path learning and flow table distribution, the hardware completes the data forwarding of the fast path. Subsequent packets are no longer sent to the SoC but are directly forwarded by the FPGA logic according to the hardware flow table.

[0110] Step S24: On the target host side, call the dpdk-devbind.py tool provided by DPDK to determine each target physical function and each target virtual function assigned to the virtual machine; use the dpdk-devbind.py tool to unload the virtio-pci driver of each target physical function and each target virtual function, and mount the vfio-pci driver, so as to change each target physical function and each target virtual function from kernel driver to user-mode driver.

[0111] In this embodiment, on the target host side, the dpdk-devbind.py tool provided by DPDK is invoked to change each PF / VF that needs to be assigned to the virtual machine from a kernel driver to a user-space driver. Specifically, the virtio-pci driver is unloaded and the vfio-pci driver is mounted, allowing the PF / VF hardware to be directly used by the virtual machine. The virtual machine bypasses the host operating system to directly control the network card hardware, eliminating the overhead caused by repeated kernel switching. Compared with existing solutions, the technical solution provided by this invention reduces the CPU consumption of the host system in network data forwarding by unloading network function hardware. This CPU resource can then be reallocated to the virtual machine Ceph service, thereby improving the overall performance of the entire virtual machine Ceph cluster storage.

[0112] Step S25: On the target host side, determine the target non-volatile fast memory assigned to the virtual machine from the target host; use the virtual file system to unload the virtio-pci driver of the target non-volatile fast memory, and mount the vfio-pci driver, so as to change the target non-volatile fast memory from a kernel driver to a user-mode driver.

[0113] In this embodiment, on the target host side, the NVMe disks to be assigned to the virtual machine are changed from kernel drivers to user-space drivers using a virtual file system (i.e., the sys file system). Specifically, the virtio-pci driver is unloaded and then the vfio-pci driver is mounted, allowing the NVMe devices to be directly accessed by the virtual machine. The virtual machine bypasses the host operating system to directly read and write to the storage disk, which is crucial for building high-performance storage servers. Compared to existing solutions that use NVMe storage disks for hardware pass-through to the virtual machine, the technical solution provided by this invention allows Ceph to bypass the host operating system and directly access data, eliminating CPU consumption and access time latency caused by kernel context switching, thereby improving the overall performance of the entire virtual machine Ceph cluster storage.

[0114] Step S26: Start each virtual machine based on the bus device function identifier corresponding to each of the target physical functions, each of the target virtual functions, and each of the target non-volatile fast memories, and deploy each virtual machine as a Ceph cluster using a preset cluster deployment tool.

[0115] In this embodiment, the BDF number of the specified PF / VF and the BDF number of the NVMe disk are used as device parameters to start the KVM virtual machine. Inside the virtual machine, the network device of type virtio-net and the storage device of type NVMe can be seen through lspci, the network interface can be seen through ip link, and the storage disk can be seen through nvme list, indicating that the device has been successfully started.

[0116] As can be seen, this application provides a method for improving network communication performance between virtual machines by offloading network applications based on smart network interface cards (NICs), increasing bandwidth and reducing latency. It also provides a method for reducing host system kernel switching overhead and accelerating access speed by using NVMe storage disks as the OSD storage medium, thereby improving the performance of Ceph cluster storage deployed on virtual machines. This deployment method can significantly reduce the host CPU's consumption in network forwarding, and more effectively apply it to virtual machine Ceph services, improving cluster storage access performance. This deployment method uses high-speed NVMe storage media directly mapped to virtual machine Ceph OSDs, greatly improving the access efficiency of the storage media, thereby improving cluster storage access performance. In other words, by applying the network offloading function provided by smart NICs, high-speed networks based on hardware forwarding are achieved; by using NVMe storage disks as the storage medium, high-performance virtual machine Ceph clusters are deployed, improving storage access performance.

[0117] See Figure 5 As shown in the figure, this application discloses a deployment apparatus for a virtual machine Ceph cluster, the apparatus comprising:

[0118] The port addition module 11 is used to run OVS-DPDK on the on-chip system of the target smart network card, create a virtual bridge, and create corresponding ports for each physical function and each virtual function that the target smart network card presents to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card pre-installed in the target host;

[0119] The driver modification module 12 is used to modify the target physical functions and target virtual functions assigned to the virtual machine from kernel drivers to user-mode drivers on the target host side, and to modify the target non-volatile fast memory in the target host assigned to the virtual machine from kernel drivers to user-mode drivers.

[0120] The cluster deployment module 13 is used to start each virtual machine based on the bus device function identifier corresponding to each of the target physical functions, each of the target virtual functions, and each of the target non-volatile fast memory, and to deploy each of the virtual machines as a Ceph cluster using a preset cluster deployment tool.

[0121] As can be seen, this application runs OVS-DPDK on the system-on-a-chip of the target smart network card, creates a virtual bridge, and creates corresponding ports for each physical function and each virtual function of the target smart network card to be presented to the target host, so as to add all the ports to the virtual bridge; wherein, the target smart network card is a network card pre-installed in the target host; on the target host side, each target physical function and each target virtual function assigned to the virtual machine is modified from kernel driver to user-mode driver, and the target non-volatile fast memory in the target host assigned to the virtual machine is modified from kernel driver to user-mode driver; each virtual machine is started based on the bus device function identifier corresponding to each target physical function, each target virtual function, and each target non-volatile fast memory, and each virtual machine is deployed as a Ceph cluster using a preset cluster deployment tool.

[0122] The beneficial effects of this application are as follows: The target smart network interface card (NIC) is pre-installed on the target host. By running OVS-DPDK on the NIC's on-chip system, creating a virtual bridge, and creating corresponding ports for each physical and virtual function of the target smart NIC exposed to the target host, all ports are added to the virtual bridge. This enables communication between physical functions, virtual functions, and physical and virtual functions on the target host side. Furthermore, on the target host side, the target physical and virtual functions assigned to the virtual machine are modified from kernel drivers to user-mode drivers, allowing the devices to be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system and directly control the NIC hardware, eliminating the overhead of repeated kernel switching. In other words, this application introduces smart NIC network hardware acceleration to improve network forwarding performance, increase network bandwidth, reduce network latency, and ultimately improve Ceph cluster storage access performance. Furthermore, on the target host side, the target non-volatile fast memory in the target host assigned to the virtual machine needs to be modified from a kernel driver to a user-mode driver so that the non-volatile fast memory device can be directly accessed by the virtual machine. Therefore, the virtual machine can bypass the host operating system to directly read and write to the storage disk, reducing the host CPU load, reducing access time, and eliminating the CPU consumption and access time latency caused by kernel context switching, thereby improving the overall performance of the entire virtual machine Ceph cluster storage. Finally, this application starts each virtual machine based on the bus device function identifiers corresponding to each target physical function, each target virtual function, and each target non-volatile fast memory, and uses a preset cluster deployment tool to deploy each virtual machine into a high-performance Ceph cluster.

[0123] In some specific embodiments, before running OVS-DPDK on the system-on-chip of the target smart network interface card, the device further includes:

[0124] A host determination unit is used to determine the physical server for deploying the Ceph cluster and to designate the physical server as the target host; wherein the physical server is equipped with a preset number of non-volatile fast memories;

[0125] A network interface card (NIC) installation unit is used to install the target smart NIC into the PCIe slot of the target host and power on the target host; wherein, the target smart NIC includes a field-programmable gate array (FPGA) and a system-on-a-chip (SoC); after power-on, the target host views the physical functions displayed to the target host by the FPGA based on the lspci command; after power-on, the SoC views the physical functions displayed to the FPGA based on the lspci command.

[0126] In some specific embodiments, the field-programmable gate array (FPGA) uses a packet forwarding method based on a fast forwarding path for packet forwarding, while the system-on-a-chip (SoC) uses a packet forwarding method based on a slow forwarding path for packet forwarding.

[0127] Accordingly, the device further includes:

[0128] A message forwarding rule configuration unit is used to pre-configure preset message forwarding rules applicable to the target communication process; the target communication process includes communication processes between different physical functions, communication processes between different virtual functions, communication processes between physical functions and virtual functions, and communication processes between physical functions and external hosts;

[0129] The preset message forwarding rules include:

[0130] If it is the first network communication packet, the first network communication packet is uploaded to the system on chip, so that the system on chip can create a corresponding flow table entry for the first network communication packet based on the slow forwarding path, and send the flow table entry and the first network communication packet to the field programmable gate array, so that the field programmable gate array can forward the first network communication packet based on the flow table entry and using the fast forwarding path;

[0131] If it is not the first network communication packet, the non-first network communication packet is uploaded to the field-programmable gate array (FPGA) so that the FPGA can forward the non-first network communication packet based on the flow table entry and using the fast forwarding path.

[0132] In some specific embodiments, the target smart network interface card includes a number of physical functions, and each physical function obtains multiple corresponding virtual functions through single root input / output virtualization technology.

[0133] In some specific embodiments, the driver modification module 12 may specifically include:

[0134] The tool invocation unit is used to invoke the dpdk-devbind.py tool provided by DPDK.

[0135] The function specification unit is used to determine the target physical functions and target virtual functions assigned to the virtual machine;

[0136] The first driver modification unit is used to uninstall the virtio-pci driver of each of the target physical functions and each of the target virtual functions using the dpdk-devbind.py tool, and mount the vfio-pci driver, so as to modify each of the target physical functions and each of the target virtual functions from kernel drivers to user-mode drivers.

[0137] In some specific embodiments, the driver modification module 12 may specifically include:

[0138] A memory designation unit is used to determine, from the target host, the target non-volatile fast memory designated for the virtual machine;

[0139] The second driver modification unit is used to unload the virtio-pci driver of the target non-volatile fast memory using a virtual file system, and mount the vfio-pci driver, so as to modify the target non-volatile fast memory from a kernel driver to a user-mode driver.

[0140] In some specific embodiments, the device further includes:

[0141] An interface determination unit is used to determine, within each of the virtual machines, the network interfaces corresponding to each of the target physical functions, each of the target virtual functions, and each of the target non-volatile fast memory, respectively.

[0142] The address configuration unit is used to configure the Internet Protocol address of the same network segment for each of the network interfaces.

[0143] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the deployment method of the virtual machine Ceph cluster executed by the electronic device disclosed in any of the foregoing embodiments.

[0144] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0145] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0146] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0147] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the deployment method of the virtual machine Ceph cluster executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0148] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the deployment method steps of the virtual machine Ceph cluster disclosed in any of the foregoing embodiments.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0152] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0153] The above provides a detailed description of the deployment method, apparatus, device, and storage medium for a Ceph virtual machine cluster provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for deploying a virtual machine Ceph cluster, characterized in that, The method comprises the following steps: running OVS-DPDK in the system on chip of the target intelligent network card, creating a virtual bridge, and creating corresponding ports for each physical function and each virtual function of the target intelligent network card exposed to the target host to add all the ports to the virtual bridge; wherein the target intelligent network card is a network card pre-installed in the target host; on the target host side, modifying each target physical function and each target virtual function assigned to a virtual machine from a kernel driver to a user state driver, and modifying a target non-volatile flash memory in the target host assigned to a virtual machine from a kernel driver to a user state driver; starting each virtual machine based on the bus device function identification number corresponding to each target physical function, each target virtual function, and each target non-volatile flash memory, and deploying each virtual machine as a Ceph cluster by using a preset cluster deployment tool.

2. The method of claim 1, wherein, Before running OVS-DPDK in the system on chip of the target intelligent network card, the method further comprises the following steps: determining a physical server for deploying a Ceph cluster, and taking the physical server as the target host; wherein the physical server is equipped with a preset number of non-volatile flash memories; installing a target intelligent network card on a PCIe slot of the target host, and powering on the target host; wherein the target intelligent network card comprises a field programmable gate array and a system on chip, and the target host views each physical function exposed to the target host by the field programmable gate array based on an lspci command after powering on; the system on chip views each physical function exposed to the system on chip by the field programmable gate array based on an lspci command after powering on.

3. The method of Claim 2, wherein, The field programmable gate array adopts a packet forwarding mode based on a fast forwarding path to perform packet forwarding, and the system on chip adopts a packet forwarding mode based on a slow forwarding path to perform packet forwarding; Correspondingly, the method further comprises the following steps: pre-configuring a preset packet forwarding rule applicable to a target communication process; the target communication process comprises a communication process between different physical functions, a communication process between different virtual functions, a communication process between the physical function and the virtual function, and a communication process between the physical function and an external host; wherein the preset packet forwarding rule comprises: if it is a first network communication packet, uploading the first network communication packet to the system on chip, so that the system on chip creates corresponding flow table items for the first network communication packet based on a slow forwarding path, and downloads the flow table items and the first network communication packet to the field programmable gate array, so that the field programmable gate array forwards the first network communication packet based on the flow table items and by using the fast forwarding path; if it is a non-first network communication packet, uploading the non-first network communication packet to the field programmable gate array, so that the field programmable gate array forwards the non-first network communication packet based on the flow table items and by using the fast forwarding path.

4. The method of claim 1, wherein, The target intelligent network card includes a plurality of physical functions, and each physical function corresponds to a plurality of virtual functions through a single root input / output virtualization technology.

5. The method of claim 1, wherein, The target physical function and the target virtual function assigned to the virtual machine are modified from a kernel driver to a user state driver, including: calling a dpdk-devbind.py tool provided by the DPDK; determining the target physical function and the target virtual function assigned to the virtual machine; uninstalling a virtio-pci driver of the target physical function and the target virtual function and mounting a vfio-pci driver to modify the target physical function and the target virtual function from the kernel driver to the user state driver by using the dpdk-devbind.py tool.

6. The method of Claim 1, wherein, The target non-volatile fast memory in the target host assigned to the virtual machine is modified from a kernel driver to a user state driver, including: determining the target non-volatile fast memory in the target host assigned to the virtual machine; uninstalling a virtio-pci driver of the target non-volatile fast memory and mounting a vfio-pci driver to modify the target non-volatile fast memory from the kernel driver to the user state driver by using a virtual file system.

7. The method of Claim 1 to 6, wherein, Further comprising: determining a network interface corresponding to each target physical function, each target virtual function and each target non-volatile fast memory in each virtual machine; configuring the same Internet Protocol address for each network interface.

8. A deployment apparatus of a virtual machine Ceph cluster, characterized in that, comprising: a port adding module, configured to run OVS-DPDK in a system on chip of a target intelligent network card, create a virtual bridge, and create corresponding ports for each physical function and each virtual function of the target intelligent network card exposed to a target host, so as to add all the ports to the virtual bridge; wherein the target intelligent network card is a network card pre-installed in the target host; a driver modifying module, configured to modify each target physical function and each target virtual function assigned to a virtual machine from a kernel driver to a user state driver on the target host side, and modify a target non-volatile fast memory in the target host assigned to the virtual machine from the kernel driver to the user state driver; a cluster deployment module, configured to start each virtual machine based on a bus device function identification number corresponding to each target physical function, each target virtual function and each target non-volatile fast memory, and deploy each virtual machine as a Ceph cluster by using a preset cluster deployment tool.

9. An electronic device, comprising: comprising: a memory, configured to store a computer program; a processor, configured to execute the computer program to implement the steps of the deployment method of the virtual machine Ceph cluster according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, for storing a computer program; wherein the computer program is executed by a processor to implement the steps of the deployment method of the virtual machine Ceph cluster according to any one of claims 1 to 7.

Images