A method and apparatus for ceph client lock optimization
By using multi-threading to maintain OSDMap resource objects in the Ceph client and utilizing task queues for communication, the performance bottleneck caused by mutex locks was resolved, achieving efficient IO processing and improving the performance and concurrency capabilities of the storage system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2023-03-13
- Publication Date
- 2026-07-31
AI Technical Summary
In multi-threaded scenarios, Ceph clients use mutex locks for synchronization, which can lead to system performance bottlenecks and affect storage system performance.
Multiple threads each maintain an OSDMap resource object, with the main thread responsible for the update process. Inter-thread communication is achieved through a task queue or an SPDK circular queue to avoid read-write locks. A hash algorithm is used to bind I/O to a specific thread to achieve lock-free processing.
It improves system concurrency, reduces I/O latency, fully utilizes cache locality, and enhances storage system performance.
Smart Images
Figure CN116303327B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of cloud computing and distributed storage technology, specifically to a Ceph client lock optimization method and apparatus. Background Technology
[0002] Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability. It is a typical client-server distributed system. During Ceph client I / O read / write operations, a read lock is acquired on the OSDMap resource object. This object contains the necessary attributes to calculate which OSD the I / O operation belongs to according to the Crush algorithm. When an OSD failure causes a change in state, the cluster's central Monitor node updates the OSDMap resource object. Ceph clients need to obtain the latest OSDMap resource object from the Monitor node when necessary to recalculate the Crush and retry the I / O. The OSDMap resource object lock is a typical read-heavy, write-light lock.
[0003] The Ceph client encapsulates each I / O operation into an Op (Operation Point). A global `opsMap` is maintained within the client process to record all ongoing Op operations. When an I / O operation needs to be committed, the Op is inserted into the Map; when an I / O operation completes, it is removed from the Map. The Map is also periodically scanned to handle Ops that time out. In traditional Ceph client solutions, mutexes are used to achieve synchronization between threads, which typically becomes a performance bottleneck in multi-threaded scenarios. Summary of the Invention
[0004] To address the aforementioned technical problems, this application provides a Ceph client lock optimization method and apparatus. Specifically, for OSDMap resource object locks in read-heavy, write-light scenarios, this application avoids acquiring read locks during I / O read / write operations by employing multiple threads each maintaining a separate OSDMap resource object. Under the same hardware resources, this reduces latency and effectively improves storage system performance. The technical solution adopted in this application is as follows:
[0005] A Ceph client lock optimization method, characterized in that the Ceph client thread pool includes multiple thread workers, and one of the multiple thread workers is designated as the main thread;
[0006] Each thread corresponds to an OSDMap resource object and can access the corresponding OSDMap resource object without locks;
[0007] When updating OSDMap resource objects, the main thread is responsible for the update process of OSDMap resource objects.
[0008] Furthermore, the main thread is responsible for the update process of the OSDMap resource object, which specifically includes: when updating the OSDMap resource object, the main thread first receives the OSDMap resource object update request, the main thread starts updating the corresponding OSDMap resource object locally, and forwards the OSDMap resource object update request to other threads to notify other threads to update the corresponding OSDMap resource object.
[0009] Furthermore, after other threads update their corresponding OSDMap resource objects, they send an update completion message to the main thread. If the main thread receives update completion messages from all other threads, the OSDMap resource object update is complete, and the main thread sends a general reply to the outside world confirming the OSDMap resource object update is complete.
[0010] Furthermore, multiple threaded workers can communicate through a task queue and event mechanism, or use the SPDK ring and SPDK polling mechanism to achieve inter-thread communication.
[0011] A Ceph client lock optimization method, wherein the Ceph client thread pool includes multiple thread workers;
[0012] When each IO is encapsulated into an Op and submitted, a HASH algorithm is used to obtain the thread ID to which the Op belongs.
[0013] The Op is submitted to the task queue of its owning thread for processing and an event notification is sent, or the owning thread can retrieve the Op by actively polling and insert it into its own opsMap for processing.
[0014] Furthermore, after each thread receives the op_reply message from the OSD, it uses the same hash algorithm based on the message's tid to obtain the thread to which the Op belongs, and then submits the task to the task queue of the thread to which it belongs. The thread to which it belongs queries its own opsMap based on the tid to obtain the Op and processes it.
[0015] Furthermore, the entire Op processing flow does not require write locks, and the sending and reply processing of the same Op is carried out in the same thread.
[0016] Furthermore, inter-thread communication can be achieved using a task queue and event notification mechanism, or by using the SPDK ring and polling mechanism.
[0017] A Ceph client lock optimization apparatus includes a processor and a memory storing processor-executable instructions, which, when executed by the processor, enable the processor to perform the method described above.
[0018] A Ceph client lock optimization apparatus includes a processor and a memory storing processor-executable instructions, which, when executed by the processor, enable the processor to perform the method described above.
[0019] The following technical effects can be achieved through the embodiments of this application:
[0020] (1) In the improved solution, the HASH algorithm is used to break down the Op and bind it to the business thread. Each thread only needs to maintain a part of the total dataset, thereby avoiding the use of mutex locks, improving the system concurrency capability and reducing IO latency.
[0021] (2) Each thread processes a relatively fixed dataset, which allows for more full utilization of the principle of cache locality, thereby achieving better performance with limited hardware resources;
[0022] (3) For read-heavy and write-light objects, each thread maintains a copy of the object, and the IO read and write process does not require read locks, thus improving the system's concurrency capabilities. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a schematic diagram of the OSDMap resource object access process in the prior art;
[0025] Figure 2 This is a schematic diagram of the Op processing flow in the prior art;
[0026] Figure 3 This is a schematic diagram of the improved OSDMap resource object update process in this application;
[0027] Figure 4 This is a schematic diagram of the improved Op processing flow of this application. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0029] This application's optimization scheme for Ceph client locks adopts a divide-and-conquer approach. It utilizes a custom load balancing mapping algorithm where each Op generates an auto-incrementing unique transaction ID. This ID is converted to a string and used as input to an FNV-1 hash function. The hash result is modulo the number of workers (threads) to determine the worker thread to which the Op belongs. After mapping the load to specific threads, the efficient lock-free queue SPDK ring inter-thread message synchronization mechanism is used to forward tasks to threads for processing. (SPDK ring is a multi-threaded, efficient programming model provided by the SPDK framework, which uses a lock-free circular queue (called a ring) to achieve efficient inter-thread message forwarding.) This application presents a general method that can be widely applied to multi-threaded, lock-free solutions in high-concurrency scenarios.
[0030] Figure 1 This diagram illustrates the existing OSDMap resource object access process. In the current Ceph client process, when handling I / O, each Ceph client thread first acquires a read lock on the OSDMap resource object, queries the object's relevant attributes, and uses the Crush algorithm to calculate the OSD node to which the I / O should be sent. When the OSD state changes, the OSDMap is updated to the cluster's central Monitor node, and the OSDMap version number (epoch) is incremented. In some cases, client threads may actively retrieve the latest OSDMap information from the Monitor and refresh the OSDMap resource object in their local memory. In this case, the updating thread first acquires a write lock and then updates the OSDMap resource object.
[0031] Figure 2 This is a schematic diagram of the existing Op processing flow. The existing Op processing flow requires a mutex lock to access the opsMap. Each thread first acquires a write lock, then inserts the Op into the global opsMap, modifies the Op's related attributes, submits it to the network thread's pending-send queue, notifies the network thread, and then releases the lock. After receiving the op_reply message from the OSD, each thread first acquires a write lock, retrieves the corresponding Op from the opsMap based on the message's transaction tid, processes it, deletes the Op from the opsMap, and finally releases the lock.
[0032] Figure 3 This is a schematic diagram of the improved OSDMap resource object update process in this application.
[0033] The Ceph client thread pool includes multiple thread workers, and one of these thread workers is designated as the master thread.
[0034] Each thread corresponds to an OSDMap resource object and can access the corresponding OSDMap resource object without locks;
[0035] When updating OSDMap resource objects, the main thread is responsible for the update process of OSDMap resource objects;
[0036] The main thread is responsible for updating OSDMap resource objects, specifically including: when updating OSDMap resource objects, the main thread first receives an update request for the OSDMap resource object, starts updating the corresponding OSDMap resource object locally, and forwards the update request to other threads, notifying them to update their corresponding OSDMap resource objects; after other threads have finished updating their corresponding OSDMap resource objects, they send an update completion message back to the main thread; if the main thread receives update completion messages from all other threads, the OSDMap resource object update is complete, and the main thread sends a general reply to the outside world that the OSDMap resource object update is complete.
[0037] Multiple threaded workers communicate with each other through a task queue and event mechanism, or through the SPDK ring and SPDK polling mechanism.
[0038] Figure 4 This is a schematic diagram of the improved Op processing flow in this application. Before multiple threads of the Ceph client concurrently send requests to the server, they first obtain an OSDMap, assemble it into an Op operation based on the information in the OSDMap, and finally send the Op operation to the server for processing (corresponding to...). Figure 4 Before the server completes the `op_submit` operation, the client stores the Op object in its Ops Map. After completing the Op operation, the server sends an ACK message back to the client. The client then processes the ACK message (corresponding to...). Figure 4 The `op_reply` method can delete the Op object or retry the Op operation based on the Op object.
[0039] In the improved Op processing flow, when each IO is encapsulated into an Op and submitted, the HASH algorithm is used to obtain the thread ID to which the Op belongs; the Op is then submitted to the task queue of its own thread for processing and an event notification is sent, or the owning thread can retrieve the Op by actively polling and insert it into its own ops Map for processing.
[0040] Subsequent processing is consistent with the original process. After receiving the op_reply message from the OSD, the business thread uses the same hash algorithm based on the message's tid to obtain the thread to which the Op belongs. Then, the task is submitted to the task queue of the thread to which the Op belongs. The thread to which the Op belongs queries its own ops Map based on the tid to obtain the Op and processes it.
[0041] The entire Op processing flow does not require write locks, and the sending and replying of the same Op are handled within the same thread. Inter-thread communication can be implemented using a task queue and event notification mechanism, or using an SPDK ring and polling mechanism.
[0042] In summary, in the technical solution of this application, each thread accesses its own OSDMap resource object. When updating the OSDMap resource object, a designated main thread is responsible for the entire update process. When the main thread receives the updated OSDMap resource object, it starts updating the object locally and notifies other threads to update. After the other threads finish updating, they will reply to the main thread with an update completion message. The main thread can reply to the outside world that the update is complete after it finds that all other threads have finished updating.
[0043] For ops map locks, the improved method uses a divide-and-conquer approach. It maps each I / O operation to a backend thread using specific mapping rules, distributing the original global ops map across multiple threads. Each thread maintains a subset of the ops map, and inter-thread communication is achieved using task queues and event notification mechanisms, or SPDK rings and polling mechanisms. Since each thread processes a relatively fixed dataset, the principle of cache locality can be fully utilized to improve program speed.
[0044] While specific embodiments of this application have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this application, but all such changes and modifications fall within the scope of protection of this application.
Claims
1. A method for Ceph client lock optimization, the method comprising: The Ceph client thread pool includes multiple thread workers, and one of these thread workers is designated as the master thread. Each thread corresponds to an OSDMap resource object and can access the corresponding OSDMap resource object without locks; When updating OSDMap resource objects, the main thread is responsible for the update process of OSDMap resource objects; The main thread is responsible for the update process of the OSDMap resource object, which specifically includes: when updating the OSDMap resource object, the main thread first receives the update request of the OSDMap resource object, the main thread starts to update the corresponding OSDMap resource object locally, and forwards the update request of the OSDMap resource object to other threads to notify other threads to update the corresponding OSDMap resource object.
2. The method of claim 1, wherein, After other threads update their corresponding OSDMap resource objects, they send an update completion message to the main thread. If the main thread receives update completion messages from all other threads, the OSDMap resource object update is complete, and the main thread sends a general reply to the outside world confirming the OSDMap resource object update is complete.
3. The method according to one of claims 1 to 2, characterized in that, Multiple threaded workers communicate with each other through a task queue and event mechanism, or through the SPDK ring and SPDK polling mechanism.
4. A method for Ceph client lock optimization, the method comprising: The Ceph client thread pool includes multiple thread workers; When each IO is encapsulated into an Op and submitted, a hash algorithm is used to obtain the thread ID to which the Op belongs. The Op is submitted to the task queue of its owning thread for processing and an event notification is sent, or the owning thread retrieves the Op by actively polling and inserts it into its own ops Map for processing. After each thread receives the op_reply message from the OSD, it uses the same hash algorithm based on the message's tid to obtain the thread to which the Op belongs. Then, it submits the Op to the task queue of the thread to which the Op belongs. The thread to which the Op belongs queries its own ops Map based on the tid and then processes the Op.
5. The method of claim 4, wherein, The entire Op processing flow does not require write locks, and the sending and reply processing of the same Op is carried out in the same thread.
6. The method according to one of claims 4 to 5, characterized in that, Inter-thread communication can be achieved using a task queue and event notification mechanism, or by using the SPDK ring and polling mechanism.
7. A Ceph client lock optimization apparatus, comprising: The device includes a processor and a memory storing processor-executable instructions, which, when executed by the processor, perform the method according to any one of claims 1 to 3.
8. A Ceph client lock optimization apparatus, comprising: The device includes a processor and a memory storing processor-executable instructions, which, when executed by the processor, perform the method according to any one of claims 4 to 6.