A processing method and system for asynchronous read IO of a Ceph OSD service end
By introducing asynchronous read I/O processing methods into Ceph OSD and utilizing the asynchronous I/O interface of the Linux kernel, the problem of worker threads being blocked due to synchronous disk I/O is solved, achieving efficient performance isolation and stable request processing, and improving the throughput and response time of the Ceph system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WINHONG INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-06-26
- Publication Date
- 2026-07-28
AI Technical Summary
The worker threads on the Ceph OSD server are blocked due to synchronous disk I/O, resulting in reduced processing capacity and impacting system performance. Existing optimization solutions have failed to completely solve this problem from an architectural perspective.
An asynchronous read I/O processing method is adopted. The read request is submitted to the kernel through the Linux kernel's asynchronous I/O interface io_submit. The kernel completes the data interaction with the underlying storage device and releases the data immediately after the worker thread initiates the asynchronous read operation. A callback function is used to process and return the data when the data is complete.
Completely freeing up OSD worker threads improves the request processing throughput and concurrency of a single OSD process, achieves performance isolation between slow disks and high-speed storage devices, reduces cluster tail latency, and improves system stability and response time.
Smart Images
Figure CN122470128A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed storage technology, and in particular to a method and system for processing asynchronous read I / O on a Ceph OSD server. Background Technology
[0002] In Ceph, the OSD (On-Site Controller) is the core service process responsible for data writing, reading, and internal data replication. When handling a client read request, the typical OSD process is: the network layer receives the request → it enters the message queue → the OSD worker thread processes it → it maps to a specific PG (placement group) → a synchronous disk read call is initiated → the thread blocks and waits for the disk to return data → the data is assembled into a network message → and sent to the client. The following are the problems with the existing technology: Synchronous disk I / O blocking OSD worker threads: This is the most fundamental problem. The number of OSD worker threads is limited (user-defined, usually related to CPU cores). When a worker thread is blocked waiting for a read operation from a slow hard drive, it cannot handle other client requests in the queue (whether these requests access other fast disks such as SSDs or other slow disks such as HDDs).
[0003] A single slow disk or slow sector can reduce the overall processing capacity of the OSD thread pool accessing that disk, causing a backlog of requests and resulting in a large number of slow OPS, thus affecting the overall performance of the system.
[0004] Limitations of existing optimization solutions: Current solutions mostly focus on avoidance (such as setting osd_op_thread_timeout to kick out requests) or early warning (such as monitoring disk latency), rather than solving the blocking problem from an architectural perspective. Increasing the number of threads introduces additional context switching overhead and complexity, and is not a fundamental solution.
[0005] Therefore, there is an urgent need in this field for a technical solution that can prevent OSD worker threads from being blocked by slow disk I / O and implement asynchronous read I / O processing to isolate the performance impact of the underlying storage device.
[0006] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention
[0007] The purpose of this invention is to provide a technical solution that can prevent OSD worker threads from being blocked by slow disk I / O and implement asynchronous read I / O processing to isolate the performance impact of the underlying storage device.
[0008] To achieve the above objectives, the present invention provides the following solution: A method for handling asynchronous read I / O on a Ceph OSD server includes: In response to a read request initiated by the client, the read request is processed by a worker thread of the OSD (Object Storage Daemon, the object storage daemon in the Ceph distributed storage system, responsible for data writing, reading, and internal replication); When it is determined that data needs to be read from the local storage device, the worker thread initiates an asynchronous read operation and registers a callback function for the asynchronous read operation; The worker thread is released and returned to the thread pool immediately after initiating the asynchronous read operation, so as to continue processing other client requests; In response to the completion of the asynchronous read operation, the read data is obtained through the callback function, and the data is assembled into a network message and sent to the client.
[0009] Furthermore, the asynchronous read operation submits the read request to the kernel by calling the Linux kernel's asynchronous I / O interface io_submit, and the kernel completes the data interaction with the underlying storage device.
[0010] Furthermore, the callback function is implemented through an asynchronous read context class, which at least stores the offset of this read operation, the data length, the target data buffer pointer, and a pointer to the original BlueStore operation context, which contains the logic for finally replying to the client.
[0011] Furthermore, the method is applied to Ceph OSD with BlueStore as the storage backend. The specific implementation steps include: adding an asynchronous read interface read_async in the BlueStore module; modifying the _do_read method of the BlueStore module so that when data needs to be read from the disk, blockdev→aread is called and the constructed asynchronous read context object is passed in, and then it returns immediately without synchronously waiting for the disk response.
[0012] Furthermore, in the step where the worker thread immediately releases and returns to the thread pool after initiating an asynchronous read operation, the worker thread does not block and wait for any disk I / O to complete. Its processing capacity is fully used to handle other client requests in the queue, including read requests to access high-speed solid-state drives and read requests to access slow mechanical hard drives.
[0013] This invention also provides a Ceph OSD server asynchronous read IO processing system, comprising: The request receiving unit is used to receive read requests initiated by the client; A worker thread pool, containing at least one OSD worker thread, the worker thread being used to process the read request and initiate an asynchronous read operation when it is determined that data needs to be read from the local storage device; An asynchronous read initiation unit is used to execute the asynchronous read operation and register a callback function for the asynchronous read operation. At the same time, it controls the worker thread to release and return to the worker thread pool immediately after initiating the asynchronous read operation. An asynchronous I / O completion processing unit is used to listen for the completion event of the asynchronous read operation and call the callback function when the operation is completed to return the read data to the client.
[0014] Furthermore, the asynchronous read operation is submitted to the kernel by calling the Linux kernel's asynchronous I / O interface io_submit, and the asynchronous I / O completion processing unit uses the kernel's asynchronous I / O completion queue or a dedicated aio monitoring thread to obtain the I / O completion status.
[0015] Furthermore, the system is completely transparent to Ceph clients, and the librbd, cephfs, or RGW components on the client side can be used compatiblely without any modifications.
[0016] Furthermore, the system operates in a heterogeneous storage device environment that includes mechanical hard drives and solid-state drives; when the read request accesses the slow mechanical hard drive, the worker thread can still be immediately released to handle other read requests accessing the fast solid-state drive, thus achieving performance isolation between the slow and fast devices.
[0017] Furthermore, the asynchronous read initiation unit and the asynchronous IO completion processing unit are integrated into the PrimaryLogPG, PGBackend and BlueStore modules of Ceph OSD. By reconstructing the IO stack, the two stages of worker thread processing requests and waiting for disk data are decoupled, so that slow disk IO is no longer a critical path that blocks the OSD service capability.
[0018] PG (Placement Group): A logical grouping unit in Ceph used to map objects to OSDs.
[0019] BlueStore: Ceph's default storage backend, directly managing raw disk devices and providing high-performance data storage.
[0020] io_submit: The Linux kernel asynchronous I / O (AIO) interface, used for batch submission of I / O requests.
[0021] aio: Asynchronous I / O, which allows a process to initiate an I / O operation without blocking and waiting for it to complete, and is notified by the kernel when the I / O is finished.
[0022] Blockdev→aread: Asynchronous read interface for block devices.
[0023] Librbd, cephfs, and RGW are Ceph's three types of client access interfaces, corresponding to block storage, file storage, and object storage, respectively.
[0024] Compared with the prior art, the present invention has the following beneficial effects: Completely liberate OSD worker threads: The core worker threads of the OSD are no longer blocked by any slow disk I / O, and their processing power is fully used to handle normal business I / O, especially the I / O of high-speed cache devices such as SSD devices, which greatly improves the request processing throughput and concurrency of a single OSD process.
[0025] Achieving fault domain isolation: The impact of slow disks is strictly limited to the asynchronous I / O commit and completion queues, preventing performance issues from spreading through the thread pool. Even if a disk responds extremely slowly, it will not significantly hinder requests to access other normal disks.
[0026] With a cache disk, slow I / O requests from upper-layer applications (such as backups and migrations) to read the HDD disk have almost no impact on normal business I / O.
[0027] During data recovery, the slow I / O of reading a large number of HDD disks has almost no impact on the performance of normal business I / O.
[0028] Reduce Ceph cluster tail latency: Since worker threads are no longer blocked, request queuing time is greatly shortened, making the response time of client read requests, especially the high percentile latency, more stable and controllable.
[0029] Fully leverage modern hardware and kernel features: Deeply integrated with the Linux kernel's high-performance asynchronous I / O mechanism, it achieves high-throughput, low-overhead I / O submission and completion processing, unleashing the full potential of high-speed storage devices such as NVMe SSDs.
[0030] Completely transparent to the client: This invention is implemented entirely on the server side, requiring zero modification to Ceph clients (such as librbd, cephfs, RGW), and is compatible with all existing applications and ecosystems. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 A flowchart for synchronous read I / O of traditional Ceph OSD.
[0033] Figure 2 The overall architecture and flowchart of the asynchronous read IO processing method of the present invention are provided for embodiments of the present invention. Detailed Implementation
[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] The purpose of this invention is to provide a technical solution that can prevent OSD worker threads from being blocked by slow disk I / O and implement asynchronous read I / O processing to isolate the performance impact of the underlying storage device.
[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0037] Example 1: A method for handling asynchronous read IO on the Ceph OSD server.
[0038] Figure 1 The traditional Ceph OSD synchronous read I / O process is illustrated, where worker threads are blocked after initiating a disk read call until data is returned before they can continue processing other requests. To solve this problem, this invention provides an asynchronous read I / O processing method, the overall architecture and process of which are as follows: Figure 2 As shown. The method described in this embodiment includes the following steps: Step 101: Receive and process client read requests.
[0039] A client (e.g., via librbd, cephfs, or RGW) sends a request to the Primary OSD where the PG resides to read an object. The OSD's network layer receives the request and places it in a message queue. A worker thread in the OSD's worker thread pool (e.g., Worker-1) retrieves the request from the queue and parses it.
[0040] Step 102: Determine the data reading path and decide to use asynchronous reading.
[0041] The worker thread maps the object information in the request to a specific Placement Group (PG) and calls the PrimaryLogPG::do_read function. This function further determines the range of data to be read from local storage devices (such as HDDs or SSDs). When it is determined that data needs to be read from the disk, the worker thread does not directly call the synchronous read interface, but instead initiates an asynchronous read operation and registers a callback function for it.
[0042] Specifically, in this embodiment, after the worker thread calls the PrimaryLogPG::do_read function, it adds the relevant read requests to the pending_async_reads context. Subsequently, the PrimaryLogPG::execute_ctx function calls... The `PrimaryLogPG::OpContext::start_async_reads` function performs asynchronous read operations.
[0043] Step 103: Call the asynchronous read interface and return immediately.
[0044] The `Start_async_reads` function further calls `pgbackend→objects_read_async`, which is ultimately executed through BlueStore's new asynchronous read interface `read_async`. The implementation of `BlueStore::read_async` is as follows: Create an AsyncReadContext object that inherits from the Context class. Its members include: the offset for this read operation, the data length (len), a pointer to the target data buffer (bufferlist *bl), and a pointer to the original BlueStore operation context (OpContext*op_ctx). The op_ctx contains the complete logic for finally replying to the client.
[0045] In the BlueStore::_do_read method, when data needs to be read from the disk, the blockdev->read synchronous read method is no longer called directly. Instead, the blockdev->aread asynchronous read method is called, and the pre-constructed AsyncReadContext object is passed in as a parameter.
[0046] The blockdev->aread layer calls the Linux kernel's asynchronous I / O interface io_submit through the KernelDevice layer to submit read requests to the kernel, which then completes the data interaction with the underlying storage device (such as an HDD or NVMe SSD).
[0047] After the above submission is completed, the worker thread (Worker-1) immediately returns from the read_async call and then to the thread pool to begin processing the next client request in the message queue. This worker thread does not block at all while waiting for disk I / O.
[0048] Step 104: Callback processing after asynchronous I / O is completed.
[0049] The KernelDevice layer has a dedicated aio monitoring thread (or utilizes the kernel's asynchronous I / O completion queue) that continuously listens for completion events of committed asynchronous read I / O. When the kernel notifies that a read I / O is complete, this thread calls a pre-registered callback function, namely the AsyncReadContext::complete method.
[0050] Inside the `AsyncReadContext::complete` method, the original BlueStore operation context is retrieved through the saved `op_ctx` pointer, thus obtaining all information associated with the original client request (including request ID, client connection, etc.). Next, the subsequent steps of data assembly and network response are executed: the data already read into the bufferlist is encapsulated into a network message and returned to the client via the OSD message sending mechanism.
[0051] At this point, a complete asynchronous read request processing is finished, and the entire process is completely transparent to the client.
[0052] Step 105: The effect of performance isolation and fault domain isolation.
[0053] Because worker threads are no longer blocked by slow disks, when heterogeneous storage devices exist in the system (such as simultaneous HDD and SSD mounting), read requests to SSDs can be processed promptly by worker threads, unaffected by other read requests waiting for HDD responses. Even if an HDD experiences slow sectors or extremely slow responses, its impact is strictly limited to the asynchronous I / O commit and completion queues and will not spread to I / O processing of other disks through the shared OSD worker thread pool. This achieves fault domain isolation and significantly reduces cluster tail latency (P99, P999).
[0054] Example 2: Asynchronous read IO processing system for Ceph OSD server.
[0055] This embodiment provides a system for implementing the above method. The system can be deployed as a software module within the Ceph OSD process, and includes the following units: Request Receiving Unit: Used to receive read requests initiated by clients. This unit interacts with the OSD's network layer and is responsible for converting raw network messages into an internal request format.
[0056] Worker thread pool: Contains at least one OSD worker thread. The worker thread is used to process the read requests and initiates an asynchronous read operation when it is determined that data needs to be read from the local storage device. The size of the worker thread pool can be configured by the user (usually related to the number of CPU cores), but this invention does not rely on increasing the number of threads to solve the blocking problem.
[0057] Asynchronous read initiation unit: This unit executes the asynchronous read operation, registers a callback function for the asynchronous read operation, and controls the worker thread to immediately release and return to the worker thread pool after initiating the asynchronous read operation. The specific implementation of this unit corresponds to the call chain PrimaryLogPG::start_async_reads, pgbackend->objects_read_async, BlueStore::read_async, and blockdev->aread described in Example 1.
[0058] Asynchronous I / O completion processing unit: This unit listens for the completion event of the asynchronous read operation and calls the callback function upon completion to return the read data to the client. This unit is implemented through the KernelDevice layer's aio monitoring thread or the kernel's asynchronous I / O completion queue, and is responsible for calling the AsyncReadContext::complete method to return the data.
[0059] In a preferred implementation of this embodiment, the asynchronous read initiation unit and the asynchronous IO completion processing unit are integrated into the PrimaryLogPG, PGBackend, and BlueStore modules of the Ceph OSD. The system is transparent to the underlying storage device type and is applicable to both hard disk drives (HDDs) and solid-state drives (SSDs / NVMe SSDs).
[0060] Example 3: Specific execution flow for a slow HDD disk scenario.
[0061] Suppose in a Ceph cluster, an OSD manages both an SSD (for caching / metadata) and an HDD (for large-capacity data storage). A client initiates two concurrent requests: request A reads a 1MB object located on the HDD, and request B reads 4KB of metadata located on the SSD.
[0062] In the conventional approach that does not employ this invention, if the worker thread processes request A first, it will synchronously wait for the HDD to return data (which may take more than 10ms). During this period, even if request B has arrived and is in the queue, it cannot be processed, resulting in the complete waste of the SSD's high performance.
[0063] When this invention is adopted, the worker thread performs the following sub-steps when processing request A: Call the asynchronous read interface and submit the HDD read request to the kernel via io_submit.
[0064] The worker thread immediately returns to the thread pool.
[0065] The worker thread retrieves request B from the queue. Since the data of request B is located on the SSD, the worker thread quickly reads 4KB of data on the SSD in a synchronous or asynchronous manner (the asynchronous mechanism of this invention can also be used), and returns the data to the client within a few microseconds.
[0066] Approximately 10ms later, the kernel completes the operation of reading 1MB of data from the HDD, and the aio monitoring thread calls the callback function to return the data to the client that requested A.
[0067] In the above process, the I / O of the slow HDD does not block the I / O processing of the fast SSD at all, achieving perfect performance isolation. This embodiment clearly demonstrates the beneficial effects of the present invention in a heterogeneous storage environment.
[0068] To achieve the above functionality, those skilled in the art need to make the following key modifications to the Ceph source code (taking BlueStore as an example): Declare the read_async interface in src / os / bluestore / BlueStore.h and implement the method in BlueStore.cc.
[0069] Define the AsyncReadContext class, usually placed in the BlueStore.cc file or a separate AsyncReadContext.h file, ensuring that it inherits from Context and implements the complete(int r) method.
[0070] Modify BlueStore::_do_read to replace synchronous read calls with conditional branches: when the caller passes a callback context, take the asynchronous path (call blockdev->aread); otherwise, keep the original synchronous path to maintain compatibility with the unmodified code.
[0071] The KernelDevice layer (src / os / bluestore / KernelDevice.cc) encapsulates aio_read, calls io_submit and io_getevents or uses the interfaces provided by libaio, and maintains a separate completion queue processing thread.
[0072] All the above modifications were completed on the Ceph OSD server side, requiring no modifications to the client side and ensuring full compatibility with the existing Ceph ecosystem.
[0073] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0074] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for handling asynchronous read I / O on a Ceph OSD server, characterized in that, include: In response to a read request initiated by the client, the read request is processed by the OSD's worker thread; When it is determined that data needs to be read from the local storage device, the worker thread initiates an asynchronous read operation and registers a callback function for the asynchronous read operation; The worker thread is released and returned to the thread pool immediately after initiating the asynchronous read operation, so as to continue processing other client requests; In response to the completion of the asynchronous read operation, the read data is obtained through the callback function, and the data is assembled into a network message and sent to the client.
2. The method for processing asynchronous read I / O on a Ceph OSD server according to claim 1, characterized in that, The asynchronous read operation submits the read request to the kernel by calling the Linux kernel's asynchronous I / O interface io_submit, and the kernel then completes the data interaction with the underlying storage device.
3. The method for processing asynchronous read I / O on a Ceph OSD server according to claim 1, characterized in that, The callback function is implemented through an asynchronous read context class, which at least stores the offset of this read operation, the data length, the target data buffer pointer, and a pointer to the original BlueStore operation context, which contains the logic for finally replying to the client.
4. The method for processing asynchronous read I / O on a Ceph OSD server according to claim 1, characterized in that, The method is applied to Ceph OSDs using BlueStore as the storage backend, and the specific implementation steps include: A new asynchronous read interface, read_async, has been added to the BlueStore module; Modify the _do_read method in the BlueStore module so that when data needs to be read from the disk, it calls blockdev→aread and passes in the constructed asynchronous read context object, and then returns immediately without synchronously waiting for a disk response.
5. The method for processing asynchronous read I / O on a Ceph OSD server according to claim 1, characterized in that, In the step where the worker thread immediately releases and returns to the thread pool after initiating an asynchronous read operation, the worker thread does not block and wait for any disk I / O to complete. Its processing capacity is fully used to handle other client requests in the queue, including read requests to access high-speed solid-state drives and read requests to access slow mechanical hard drives.
6. A Ceph OSD server-side asynchronous read I / O processing system, characterized in that, include: The request receiving unit is used to receive read requests initiated by the client; A worker thread pool, containing at least one OSD worker thread, the worker thread being used to process the read request and initiate an asynchronous read operation when it is determined that data needs to be read from the local storage device; An asynchronous read initiation unit is used to execute the asynchronous read operation and register a callback function for the asynchronous read operation. At the same time, it controls the worker thread to release and return to the worker thread pool immediately after initiating the asynchronous read operation. An asynchronous I / O completion processing unit is used to listen for the completion event of the asynchronous read operation and call the callback function when the operation is completed to return the read data to the client.
7. The Ceph OSD server asynchronous read IO processing system according to claim 6, characterized in that, The asynchronous read operation is submitted to the kernel by calling the Linux kernel's asynchronous I / O interface io_submit. The asynchronous I / O completion processing unit uses the kernel's asynchronous I / O completion queue or a dedicated aio monitoring thread to obtain the I / O completion status.
8. The Ceph OSD server asynchronous read IO processing system according to claim 6, characterized in that, The system is completely transparent to Ceph clients, and the librbd, cephfs, or RGW components on the client side can be used compatiblely without any modifications.
9. The Ceph OSD server asynchronous read IO processing system according to claim 6, characterized in that, The system operates in a heterogeneous storage environment that includes mechanical hard drives and solid-state drives; when a read request accesses a slow mechanical hard drive, the worker thread can still be immediately released to handle other read requests accessing a fast solid-state drive, thus achieving performance isolation between slow and fast devices.
10. The Ceph OSD server asynchronous read IO processing system according to claim 6, characterized in that, The asynchronous read initiation unit and asynchronous IO completion processing unit are integrated into the PrimaryLogPG, PGBackend and BlueStore modules of Ceph OSD. By reconstructing the IO stack, the two stages of worker thread processing requests and waiting for disk data are decoupled, so that slow disk IO is no longer a critical path that blocks the OSD service capability.