Method and device for improving speed of SPDK disk IO
By splitting the vhost target layer and the bdev device driver layer and creating a thread pool for parallel processing, the problem of low processing efficiency in the existing SPDK IO model is solved, and a significant improvement in virtual machine IO performance is achieved.
Patent Information
- Application Number
- CN202310180458.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-24
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2043-02-24
AI Technical Summary
In the existing SPDK IO model, there are multiple device queues inside the virtual machine, but only one vhost thread handles IO requests, resulting in long processing time at the bdev layer and low overall process efficiency.
The vhost target layer and the bdev device driver layer are separated, and vhost thread pools and bdev thread pools are created to handle them separately. The threads in the vhost thread pool share memory with the virtual machine and interact to transform I/O requests, while the bdev thread pool handles I/O requests based on a preset strategy.
By using a multi-threaded processing mode, the I/O performance of the virtual machine is improved, the processing efficiency is optimized, and parallel processing of multi-queue disk devices is achieved, avoiding the waiting time of single-threaded processing.
Smart Images
Figure CN116027987B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of cloud computing and data storage, and particularly relates to a method and device for improving the speed of SPDK disk IO. BACKGROUND
[0002] SPDK is a storage development kit, which is specially designed for special devices (such as NVMe SSD, etc.), and provides a series of high-performance, scalable, user-space tools and libraries, and is currently widely used in the field of cloud computing. SPDK has three advantages: 1. The driving code of the storage device runs in the user state, rather than in the kernel state, avoiding the context switching between the kernel and the user state, and saving the CPU (Central Processing Unit) overhead; 2. Polling mode driving, changing the IO (Input / Output) interrupt processing mode, the IO request will be responded immediately after being submitted, rather than waiting for the interrupt to wake up, reducing the delay; 3. Avoiding the use of locks in the IO path, and using the message mechanism, thereby avoiding the overhead of locks.
[0003] SPDK is divided into three levels, the uppermost layer is the storage protocol layer, which exposes different protocol interfaces such as iscsi target, vhosttarget, nvme-of target, etc. to the outside; the middle layer is the storage service layer, which mainly provides specific storage services, and this layer includes a block device abstraction layer, and various specific drivers below the block device abstraction layer, such as malloc (dynamic memory allocation), ceph rbd, aio, etc.; the bottom layer is the specific storage device driver layer, such as nvme device driver, etc.
[0004] For the IO model of SPDK, taking vhost target (used to accelerate IO processing) as an example: the virtual machine OS (operating system) applies a piece of memory as shared memory, which is organized in the form of virtio queue (a kind of queue form), and stores IO requests and IO completion information, etc., and the virtual machine process and the vhost process (the carrier process of SPDK function) transfer IO information through this piece of memory, and the virtio queue corresponds to the queue of the internal device driver of the virtual machine one by one. Referring to Figure 1 As shown, the flow of IO processing is as follows: 1. The OS system in the virtual machine puts the IO request in the form of virtio ring (a kind of queue) into the shared memory queue; 2. The vhost thread gets the IO request from the shared memory queue, and then converts the request into a Bdev layer (SPDK block device layer) request, and then converts it into an operation on the real device through the specific device driver of the bdev layer.
[0005] It can be obviously seen that the existing IO model has the following problems: 1, there are multiple device queues in a device in the virtual machine, that is, there are multiple virtio queues, but only one vhost thread obtains IO requests from multiple shared queues for processing; 2, the processing of the bdev layer is in the same thread as the vhost target layer, if the back-end device is a slow device, the bdev layer processing time will be longer, resulting in longer processing time of the entire process, even if the vhost target time is very short, it also needs to wait for the slow device to process, resulting in low processing efficiency of the entire process. SUMMARY
[0006] In view of the defects in the prior art, the purpose of the present application is to provide a method and device for improving the speed of SPDK disk IO, which can greatly improve the IO performance of the virtual machine, and the optimization does not modify the operation method of the vhost target, thereby effectively improving the processing efficiency.
[0007] To achieve the above purpose, the present application provides a method for improving the speed of SPDK disk IO, which specifically comprises the following steps:
[0008] Split the vhost target layer and the bdev device driver layer, and create a vhost thread pool and a bdev thread pool;
[0009] The vhost target layer is processed based on the threads of the vhost thread pool, and the bdev device driver layer is processed based on the threads of the bdev thread pool.
[0010] On the basis of the above technical solution,
[0011] The threads of the vhost thread pool are used to interact with the shared memory of the virtual machine, obtain IO requests from the shared memory or put IO requests into the shared memory;
[0012] The threads of the vhost thread pool are also used to convert the IO requests into requests required by the bdev device driver layer, and send the converted IO requests to the threads of the bdev thread pool for processing.
[0013] On the basis of the above technical solution, the bdev thread pool selects threads based on a preset strategy to process the converted IO requests.
[0014] On the basis of the above technical solution, the vhost thread pool is in a multi-thread mode, when the virtual machine is configured with multiple queues, the queues are evenly distributed on the threads of the vhost thread pool.
[0015] On the basis of the above technical solutions, the number of threads in the bdev thread pool is self-defined, and the number of threads in the bdev thread pool is 0 or a positive integer.
[0016] The application provides a device for improving the speed of a SPDK disk IO.
[0017] A creating module is configured to split the vhost target layer and the bdev device driver layer, and create a vhost thread pool and a bdev thread pool.
[0018] A processing module is configured to process the vhost target layer based on the threads of the vhost thread pool, and process the bdev device driver layer based on the threads of the bdev thread pool.
[0019] On the basis of the above technical solutions,
[0020] The threads of the vhost thread pool are configured to interact with the shared memory of the virtual machine, obtain IO requests from the shared memory, or put IO requests into the shared memory.
[0021] The threads of the vhost thread pool are further configured to convert the IO requests into requests required by the bdev device driver layer, and send the converted IO requests to the threads of the bdev thread pool for processing.
[0022] On the basis of the above technical solutions, the bdev thread pool selects threads based on a preset strategy to process the converted IO requests.
[0023] On the basis of the above technical solutions, the vhost thread pool is in a multi-thread mode, and when the virtual machine is configured with multiple queues, the queues are uniformly distributed on the threads of the vhost thread pool.
[0024] On the basis of the above technical solutions, the number of threads in the bdev thread pool is self-defined, and the number of threads in the bdev thread pool is 0 or a positive integer.
[0025] Compared with the prior art, the application has the advantages that: by creating a vhost thread pool and a bdev thread pool to respectively process the vhost target layer and the bdev device driver layer, the multi-queue disk device of the virtual machine can be processed by multiple threads of the vhost thread pool at the same time instead of only one thread, and after the thread of the vhost thread pool obtains an IO request, the thread submits the IO request to the bdev thread pool for processing without waiting for the IO request to be sent to the real device, so that the thread of the vhost thread pool can continue to obtain IO requests from the shared queue for processing, thereby greatly optimizing and improving the IO performance of the virtual machine, and the optimization does not modify the operation method of the vhost target, and can effectively improve the processing efficiency. BRIEF DESCRIPTION OF DRAWINGS
[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0027] Figure 1 An existing IO processing flowchart;
[0028] Figure 2 A flowchart of a method for improving the SPDK disk IO speed in the embodiment of the present application;
[0029] Figure 3 An IO processing flowchart of the present application;
[0030] Figure 4 An IO processing flowchart when the number of threads of the bdev thread pool is configured as 0. DETAILED DESCRIPTION
[0031] This invention provides a method to improve SPDK disk I / O speed. By creating a vhost thread pool and a bdev thread pool to handle vhost target layer processing and bdev device driver layer processing respectively, the virtual machine's multi-queue configured disk device can be processed simultaneously by multiple threads in the vhost thread pool, instead of being processed by only one thread. Furthermore, after a vhost thread pool thread receives an I / O request, it submits it to the bdev thread pool for processing. It does not need to wait for the I / O request to be sent to the actual device; the vhost thread pool thread can continue to retrieve I / O requests from the shared queue for processing. This significantly optimizes and improves virtual machine I / O performance. Moreover, this optimization does not modify the vhost target operation method, effectively improving processing efficiency. This invention also provides a device for improving SPDK disk I / O speed.
[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.
[0033] See Figure 2 As shown in the figure, an embodiment of the present invention provides a method for improving the disk I / O speed of SPDK, which specifically includes the following steps:
[0034] S1: Separate the vhost target layer and the bdev device driver layer, and create vhost thread pools and bdev thread pools;
[0035] S2: Threads based on the vhost thread pool handle vhost target layer processing, while threads based on the bdev thread pool handle bdev device driver layer processing. For example... Figure 3 As shown, this invention changes the vhost from a single thread to a multi-threaded process, and the IO processing flow, which was originally completed by one thread, is divided into two stages, each completed by a corresponding thread.
[0036] In this invention, the threads of the vhost thread pool are used to interact with the shared memory of the virtual machine, to obtain or put IO requests into the shared memory; the threads of the vhost thread pool are also used to convert IO requests into requests required by the bdev device driver layer, and send the converted IO requests to the threads of the bdev thread pool for processing.
[0037] In this invention, the bdev thread pool selects threads to process the converted IO requests based on a preset strategy. That is, the threads for processing the converted IO requests can be customized, such as a certain thread being used to process a certain type of converted IO request, etc.
[0038] In the present application, the vhost thread pool is a multi-thread mode, when the virtual machine is configured with multiple queues, the queues are evenly distributed to the threads of the vhost thread pool, that is, each thread is used to process a queue.
[0039] It should be noted that the number of threads in the bdev thread pool is self-defined, and the number of threads in the bdev thread pool is 0 or a positive integer. That is, the number of threads in the bdev thread pool can be configured according to the actual situation, or it can be configured to 0. For local devices, the two-layer thread pool affects the speed, so it can be changed to the form as shown in Figure 4 Each thread in the vhost thread pool completes the processing of the IO request until it is submitted to the real physical device.
[0040] It should be noted that, Figure 3 In the SPDK process box, the other part outside the box is the virtual machine process, including the shared memory multi-queue, which is the memory applied by the virtual machine, Figure 2 In the virtual machine, the virtio blk disk is presented, if the vhost scsi controller is added to the SPDK, the virtio scsi disk is presented in the virtual machine. The SPDK process includes two parts of the vhost thread pool and the bdev thread pool, and also includes the underlying device driver and the device.
[0041] The method for improving the SPDK disk IO speed of the present application is specifically described below by taking the vhost user blk target as an example:
[0042] A: Start the vhost target process, and execute the code:
[0043] build / bin / vhost-S / var / tmp-m 0x3;
[0044] Here, -m is the CPU mask bound to the vhost thread, which is bound to 2 CPUs, and 2 vhost threads will be started accordingly;
[0045] B: Execute the code: scripts / rpc.py bdev_malloc_create 64 512-b Malloc0;
[0046] This step creates a bdev disk device, and the newly added function is to create a bdev-related thread pool to receive the IO request of the vhost thread;
[0047] C: Execute code: scripts / rpc.py vhost_create_blk_controller --cpumask 0x3 vhost.1 Malloc 0;
[0048] That is, create a vhost blk disk controller, here cpumask binds the CPU list of the disk IO request processing, and the corresponding thread on the CPU obtains the IO request from the shared queue;
[0049] D: Start the virtual machine. The disk corresponds to the libvirt xml information as follows:
[0050] <disk type='vhostuser' device='disk'>
[0051] <driver name='qemu' type='raw' queues='4' / >
[0052] <source type＝'unix'path＝' var tmp vhost.1'>
[0053]
[0054] <target dev='vdb' bus='virtio' / >
[0055]
[0056] queues is the number of multi-queue, and SPDK will evenly distribute these queues to the CPU list bound in step C.
[0057] In the optimization of the IO model of SPDK, the original interface of SPDK is not modified, and the original interface parameters are completely used for adaptation to complete the multi-thread function; the application changes the working mode of the single-thread processing disk of SPDK, and greatly improves the virtual machine disk IO speed and performance.
[0058] In the application, the bdev device driver layer adopts malloc device, the vhost target layer adopts vhost-user-blk controller, the virtual machine SPDK disk adopts 4 queues, the vhost thread is bound to 2 CPUs, the fio is executed in the virtual machine to perform random reading, and the bs size is 4K. Before optimization, the iops is 30K / s, and after optimization, the iops reaches 80K / s, which has reached the upper limit of the speed of the spdk malloc bdev device.
[0059] The application modifies the single-thread IO processing model of SPDK into a multi-stage and multi-thread processing mode, which is originally processed by a vhost thread from shared memory data acquisition to the bottom device, and is changed into a two-stage processing mode of vhost thread and bdev device driver layer thread; the existing SPDK interface parameter-m is used to establish a corresponding number of vhost processing threads; the vhost multi-thread simultaneously processes the allocation strategy of the shared memory multi-queue of the virtual machine, which is not limited to the equal division strategy according to the number of queues, but also includes the busy degree method; the application describes the vhost target, and other spdk targets, such as nvme-of target, iscsi target and the like, also have the same problem, and the method of the application is also applicable; the thread pool of the bdev layer can be adjusted according to the actual situation, and the number of threads can be configured, or it can be configured as 0, that is, the processing of the bdev layer is also processed in the vhost thread, only the vhost multi-thread mode is saved.
[0060] The method for improving the SPDK disk IO speed of the embodiment of the application creates a vhost thread pool and a bdev thread pool to respectively process the vhost target layer and the bdev device driver layer, so that the disk device configured with multiple queues of the virtual machine can be simultaneously processed by multiple threads of the vhost thread pool instead of only being processed by one thread, and meanwhile, after the thread of the vhost thread pool obtains an IO request, the thread of the bdev thread pool processes the IO request, and the thread of the vhost thread pool can continue to obtain the IO request from the shared queue for processing without waiting for the IO request to be issued to the real device, thereby greatly optimizing and improving the IO performance of the virtual machine, and the optimization does not modify the operation method of the vhost target, and can effectively improve the processing efficiency.
[0061] In a possible implementation, the embodiment of the application further provides a readable storage medium located in a PLC (Programmable Logic Controller, programmable logic controller) controller, and a computer program is stored on the readable storage medium, and the program is executed by a processor to implement the steps of the method for improving the SPDK disk IO speed as follows:
[0062] The vhost target layer and the bdev device driver layer are split, and a vhost thread pool and a bdev thread pool are created;
[0063] The vhost target layer is processed based on the thread of the vhost thread pool, and the bdev device driver layer is processed based on the thread of the bdev thread pool.
[0064] The storage medium can include any one or more of a computer-readable medium or a computer-readable storage medium. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, the computer-readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
[0065] A computer-readable signal medium can include a computer-readable instruction, a data signal, a program code in a baseband, or a carrier wave that propagates through a transmission medium, in which the computer-readable instruction, the data signal, the program code, or the carrier wave can be used by or in connection with an instruction execution system, apparatus, or device. The computer-readable signal medium also can be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable signal medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0066] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, application specific circuitry, or field programmable gate array (FPGA) circuitry can execute the computer program code.
[0067] The device for improving the SPDK disk IO speed provided by the embodiment of the application comprises a creating module and a processing module. The creating module is used for splitting a vhost target layer and a bdev device driver layer, and creating a vhost thread pool and a bdev thread pool. The processing module is used for processing the vhost target layer based on a thread of the vhost thread pool, and processing the bdev device driver layer based on a thread of the bdev thread pool.
[0068] In the application, the thread of the vhost thread pool is used for interacting with a shared memory of a virtual machine, obtaining or putting an IO request into the shared memory; the thread of the vhost thread pool is also used for converting the IO request into a required request of the bdev device driver layer, and sending the converted IO request to the thread of the bdev thread pool for processing.
[0069] In the application, the bdev thread pool selects a thread based on a preset strategy to process the converted IO request.
[0070] In the application, the vhost thread pool is in a multi-thread mode, and when the virtual machine is configured with multiple queues, the queues are evenly distributed on the threads of the vhost thread pool.
[0071] In the application, the number of threads in the bdev thread pool is self-defined, and the number of threads in the bdev thread pool is 0 or a positive integer.
[0072] The device for improving the SPDK disk IO speed of the embodiment of the application creates a vhost thread pool and a bdev thread pool to process the vhost target layer and the bdev device driver layer respectively, so that the disk device configured with multiple queues of the virtual machine can be processed by multiple threads of the vhost thread pool at the same time, instead of being processed by only one thread. Meanwhile, after the thread of the vhost thread pool obtains the IO request, the thread of the vhost thread pool submits the IO request to the bdev thread pool for processing, and does not need to wait for the IO request to be sent to a real device. The thread of the vhost thread pool can continue to obtain the IO request from the shared queue for processing, thereby greatly optimizing and improving the IO performance of the virtual machine. Moreover, the optimization does not modify the operation method of the vhost target, and can effectively improve the processing efficiency.
[0073] The above is only the specific embodiment of the application, so that the person skilled in the art can understand or implement the application. Various modifications of the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.
[0074] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the variations can describe certain implementation thereof as performed by one or more computer systems, e.g., computer systems 1000. However, the computer systems need not be limited to such computer systems, and additional or fewer computer systems can be utilized. For example, one or more computer systems can be utilized by various entities, such as a client, a server, a client and a server, etc. In one implementation, the computer systems can be implemented using one or more virtual machines running on one or more physical computer systems. Figure 1 The flowchart and / or block diagram in the variations can describe certain implementation thereof as performed by one or more computer systems, e.g., computer systems 1000. However, the computer systems need not be limited to such computer systems, and additional or fewer computer systems can be utilized. For example, one or more computer systems can be utilized by various entities, such as a client, a server, a client and a server, etc. In one implementation, the computer systems can be implemented using one or more virtual machines running on one or more physical computer systems.
Claims
1. A method for improving SPDK disk I / O speed, characterized in that, Specifically, the following steps are included: The vhost target layer and bdev device driver layer are separated, and vhost thread pools and bdev thread pools are created. Threads based on the vhost thread pool are used for processing the vhost target layer, and threads based on the bdev thread pool are used for processing the bdev device driver layer. The vhost thread pool is in multi-threaded mode. When the virtual machine is configured with multiple queues, the queues are evenly distributed among the threads of the vhost thread pool. in, The threads in the vhost thread pool are used to interact with the virtual machine's shared memory, obtaining or putting IO requests into the shared memory. The threads in the vhost thread pool are also used to convert IO requests into requests required by the bdev device driver layer, and send the converted IO requests to the threads in the bdev thread pool for processing. The number of threads in the bdev thread pool can be customized, and the number of threads in the bdev thread pool is a positive integer.
2. The method for improving SPDK disk I / O speed as described in claim 1, characterized in that: The bdev thread pool selects threads based on a preset strategy to process the transformed IO requests.
3. An apparatus for improving the disk I / O speed of SPDK, characterized in that, include: Create a module that splits the vhost target layer and the bdev device driver layer, and creates vhost thread pools and bdev thread pools; The processing module is used to process the vhost target layer based on the vhost thread pool and to process the bdev device driver layer based on the bdev thread pool. The vhost thread pool is in multi-threaded mode. When the virtual machine is configured with multiple queues, the queues are evenly distributed among the threads of the vhost thread pool. in, The threads in the vhost thread pool are used to interact with the virtual machine's shared memory, obtaining or putting IO requests into the shared memory. The threads in the vhost thread pool are also used to convert IO requests into requests required by the bdev device driver layer, and send the converted IO requests to the threads in the bdev thread pool for processing. The number of threads in the bdev thread pool can be customized, and the number of threads in the bdev thread pool is a positive integer.
4. The apparatus for improving SPDK disk I / O speed as described in claim 3, characterized in that: The bdev thread pool selects threads based on a preset strategy to process the transformed IO requests.