Ceph performance tuning method
By dynamically adjusting the binding strategy between Ceph-osd threads and CPU cores, the problems of resource contention and load imbalance caused by improper thread scheduling are solved, thereby improving the reliability and performance of the Ceph storage system.
Patent Information
- Application Number
- CN202510866009.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-26
- Publication Date
- 2025-10-17
AI Technical Summary
In Ceph systems, improper thread scheduling leads to CPU resource contention, memory access delays, and load imbalance, affecting system performance.
By adding a performance tuning daemon, the binding strategy between Ceph-osd threads and CPU cores is dynamically adjusted. The htop and taskset tools are used to monitor and bind threads to a specified CPU set in real time, thus avoiding resource contention between threads.
Improves the reliability and stability of the Ceph storage system, solves the problem of high CPU consumption when ceph-osd empty snapshot snaptrim, and improves performance.
Smart Images
Figure CN120803702A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application discloses a Ceph performance tuning method and relates to the technical field of cloud computing distributed storage. BACKGROUND
[0002] With the wide application of large-scale distributed storage systems, Ceph storage systems have attracted widespread attention due to their high scalability and high performance characteristics. Ceph is an open-source distributed storage system that can provide block storage, object storage and file system storage and other services. Due to its excellent flexibility and expansion capability, Ceph has become one of the mainstream choices in the field of cloud computing and data centers.
[0003] In the Ceph system, there are multiple types of threads that are responsible for different tasks, such as business processing threads: read-write operations, data replication, heartbeat detection threads, network data transceiver threads, command interaction threads, etc. The performance of these threads directly affects the performance of the entire system. In a multi-core architecture, if the thread scheduling is not proper, it may cause some problems, such as:
[0004] CPU resource contention: multiple threads competing for the same CPU resources can cause frequent context switching, increasing system overhead.
[0005] Memory access delay: when threads migrate between different CPU cores, it may increase memory access delay, affecting performance.
[0006] Load imbalance: uneven distribution of threads on various CPU cores can cause some cores to be excessively busy, while other cores have low utilization. SUMMARY
[0007] The application provides a Ceph performance tuning method to solve the problem of excessive CPU consumption caused by ceph-osd snaptrim, improve the reliability and stability of the entire ceph cluster, and improve the performance of the Ceph storage system.
[0008] The specific scheme provided by the application is:
[0009] The application provides a Ceph performance tuning method, comprising:
[0010] Step 1: by analyzing the Ceph-osd system source code, key business thread model and runtime monitoring, the threads of Ceph-osd are divided into business processing threads, network message transceiver threads, message distribution threads, heartbeat threads, command interaction threads and other threads,
[0011] Step 2: Add a performance tuning daemon to monitor the startup and exit of the storage node osd process in real time, obtain the thread number of the Ceph-osd thread through the htop tool, and bind the Ceph-osd thread to the specified CPU set according to the configuration item configuration through the taskset tool to avoid thread resource preemption.
[0012] Further, the service processing thread in the Ceph-osd thread involved in step 1 of the Ceph performance tuning method is responsible for all key service processing, including data read and write, data recovery,
[0013] The network message receiving thread involved is responsible for the network communication of Ceph-osd, receives external network data, and sends network data to the opposite end,
[0014] The message distribution thread involved is responsible for converting the network data received by the network message receiving thread into internal Ceph messages and forwarding them to the service processing thread for processing,
[0015] The heartbeat thread involved is responsible for sending heartbeat detection messages to peerosd,
[0016] The osd command interaction thread involved is responsible for processing instructions issued by the outside to the osd for configuration changes.
[0017] Further, in step 2 of the Ceph performance tuning method, it specifically includes:
[0018] Step 21: Monitor the startup and exit of the osd process in real time through the performance tuning daemon:
[0019] Add a notification script when the ceph-osd service starts and exits to inform the performance daemon of the change in the ceph-osd process, and set and cancel the CPU binding core through taskset,
[0020] Step 22: Obtain the thread number through the performance tuning daemon:
[0021] When a new ceph-osd process is started, obtain the core thread number of the ceph-osd service through the performance tuning daemon according to the htop command and record it, and obtain the thread number through the htop tool according to the thread name of the five types of threads of Ceph-osd,
[0022] Step 23: Bind the CPU set of the thread through the performance tuning daemon using taskset:
[0023] In order to avoid the CPU resource preemption among threads, five categories of threads are bound to different CPU cores respectively, which are determined by the following five configuration items: osd_op_tp_cpuset configuration item, msgr-worker_cpuset configuration item, ms_dispatch_cpuset configuration item, osd_srv_heartbt_cpuset configuration item and tp_osd_cmd_cpuset configuration item.
[0024] The CPU set of the thread is dynamically set by the performance daemon through the taskset command, specifically through taskset -c CPU set -p thread number.
[0025] Step 24: When the ceph-osd process exits, the thread binding core is cleaned up by the performance tuning daemon:
[0026] When the ceph-osd process exits, the performance tuning daemon cancels the CPU binding core of the thread according to the relationship between the osd process and the thread number recorded in the memory.
[0027] Further, in step 2 of the Ceph performance tuning method, for other threads of the ceph-osd that are not bound to the CPU core, the operating system and other services recommend using the remaining preset share of CPU resources or preoccupying the CPU resources allocated above.
[0028] The application provides a Ceph performance tuning device, which comprises a classification module, an osd process timing detection module, an adding module, an osd core thread number acquisition module, a thread binding CPU set module,
[0029] The classification module divides the threads of the Ceph-osd into business processing threads, network message transmission threads, message distribution threads, heartbeat threads, command interaction threads and other threads by analyzing the Ceph-osd system source code, the key business thread model and the runtime monitoring situation,
[0030] The adding module adds a performance tuning daemon, the sd process timing detection module monitors the startup and exit of the storage node osd process in real time, the osd core thread number acquisition module acquires the thread number of the Ceph-osd thread through the htop tool, and the thread binding CPU set module binds the Ceph-osd thread to the specified CPU set according to the configuration of the configuration item through the taskset tool, so as to avoid thread resource preemption.
[0031] Further, the classification module of the Ceph performance tuning device is related to the business processing thread in the Ceph-osd thread, which is responsible for all key business processing, including data reading and writing, data recovery,
[0032] The network message sending and receiving threads involved are responsible for Ceph-osd's network communication, receiving external network data, and sending network data to the other end.
[0033] The message distribution thread involved is responsible for converting the network data received by the network message sending and receiving thread into Ceph internal messages and forwarding them to the business processing thread for processing.
[0034] The heartbeat thread involved is responsible for sending heartbeat detection messages to peerosd.
[0035] The OSD command interaction thread involved is responsible for processing external instructions sent to OSD for configuration changes.
[0036] Furthermore, the osd process timing detection module of the Ceph performance tuning device monitors the startup and exit of the osd process in real time through the performance tuning daemon process:
[0037] Add notification scripts when the ceph-osd service is started and exited to inform the performance daemon ceph-osd process of changes, set the CPU set and cancel the CPU binding core through taskset,
[0038] The osd core thread number acquisition module obtains the thread number through the performance tuning daemon:
[0039] When the ceph-osd process is newly started, the performance tuning daemon obtains the core thread number of the ceph-osd service according to the htop command and records it. According to the thread names of the five major types of Ceph-osd threads, the thread number is obtained through the htop tool.
[0040] The thread-binding CPU set module uses taskset to bind the CPU set of threads through the performance tuning daemon:
[0041] To avoid CPU resource preemption between threads, the five major types of threads are bound to different CPU cores, which are determined by the following five configuration items: osd_op_tp_cpuset configuration item, msgr-worker_cpuset configuration item, ms_dispatch_cpuset configuration item, osd_srv_heartbt_cpuset configuration item, and tp_osd_cmd_cpuset configuration item;
[0042] The CPU set of the thread is dynamically set through the taskset command through the performance daemon, specifically through taskset -c CPU set -p thread number.
[0043] When the ceph-osd process exits, the thread binding CPU set module cleans up the thread binding core through the performance tuning daemon:
[0044] When the ceph-osd process exits, the thread binding CPU set module cleans up the thread binding core through the performance tuning daemon:
[0045] Further, the thread binding CPU set module of the Ceph performance tuning device recommends using the remaining preset share of CPU resources or preoccupies the CPU resources allocated for other threads of the ceph-osd, the operating system and other services that are not bound to the CPU core.
[0046] The present application has the advantages that:
[0047] The present application adds a performance tuning daemon and a configuration file, does not intrude on the ceph-osd program, belongs to an external plug-in timing extraction mechanism, can bind threads of different categories to corresponding CPU sets during the running of the ceph-osd, optimizes the binding strategy between threads and CPU cores, solves the problem of ceph-osd down caused by the large consumption of CPU during the ceph-osd empty snapshot snaptrim, improves the reliability and stability of the entire ceph cluster, and improves the performance of the Ceph storage system. BRIEF DESCRIPTION OF DRAWINGS
[0048] Figure 1 is a method flowchart of the present application.
[0049] Figure 2 is a core binding process flowchart of the ceph-osd process.
[0050] Figure 3 is a core unbinding process flowchart of the ceph-osd process. DETAILED DESCRIPTION
[0051] The present application will be further described below in combination with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting to the present application.
[0052] Embodiment 1
[0053] The present application provides a Ceph performance tuning method, comprising:
[0054] Step 1: By analyzing the Ceph-osd system source code, the key business thread model and the runtime monitoring situation, the threads of the Ceph-osd are divided into business processing threads, network message receiving and sending threads, message distribution threads, heartbeat threads, command interaction threads and other threads.
[0055] Among them, the business processing thread, that is, the osd_op_tp thread is the most core thread of Ceph-osd, responsible for all key business processing, including data reading and writing, data recovery, etc., and is the most busy thread of Ceph-osd and also the thread with the largest concurrency. For the data disk of hdd medium, the osd has 5 osd_op_tp threads by default to execute concurrently; for the data disk of ssd medium, the osd has 8 osd_op_tp threads by default to execute concurrently.
[0056] The network message receiving thread, that is, the msgr-worker thread, is responsible for the network communication of Ceph-osd, receives external network data, and sends network data to the opposite end, and the osd has 3 msgr-worker threads by default.
[0057] The message distribution thread, that is, the ms_dispatch thread, is responsible for converting the network data accepted by the msgr-worker thread into internal messages of Ceph: read and write requests, heartbeat messages, etc., and forwarding them to the osd_op_tp thread for processing, and the osd has 5 ms_dispatch threads.
[0058] The heartbeat thread, that is, the osd_srv_heartbt thread, is responsible for sending heartbeat detection messages to peer osd, and the osd has 1 thread.
[0059] The command interaction thread, that is, the tp_osd_cmd thread, is responsible for processing the instructions issued by the outside to the osd, and is mainly used for configuration changes, and the osd has 1 thread.
[0060] Other threads include finisher, safe_time, etc.
[0061] Step 2: Add a performance tuning daemon to monitor the startup and exit of the osd process of the storage node in real time, obtain the thread number of the Ceph-osd thread through the htop tool, and bind the Ceph-osd thread to the specified CPU set according to the configuration item configuration through the taskset tool to avoid thread resource preemption.
[0062] Specifically, it includes:
[0063] Step 21: Monitor the startup and exit of the osd process in real time through the performance tuning daemon:
[0064] When the ceph-osd service starts and exits, a notification script is added respectively to inform the performance daemon that the ceph-osd process has changed, and the CPU set is set and the CPU binding core is canceled through taskset,
[0065] Step 22: Obtain thread number by performance tuning daemon:
[0066] When a new ceph-osd process is started, the core thread number of the ceph-osd service is obtained by the performance tuning daemon according to the htop command, and is recorded according to the thread name of the five categories of Ceph-osd threads, and the thread number is obtained by the htop tool,
[0067] Step 23: Bind the CPU set of the thread by the performance tuning daemon using taskset:
[0068] In order to avoid CPU resource preemption between threads, the five categories of threads are respectively bound to different CPU cores, which are respectively determined by the following five configuration items: osd_op_tp_cpuset configuration item, msgr-worker_cpuset configuration item, ms_dispatch_cpuset configuration item, osd_srv_heartbt_cpuset configuration item and tp_osd_cmd_cpuset configuration item.
[0069] The CPU set of the thread is dynamically set by the performance daemon through the taskset command, specifically through taskset-c CPU set-p thread number.
[0070] Step 24: When the ceph-osd process exits, the performance tuning daemon cleans up the thread binding core:
[0071] When the ceph-osd process exits, the performance tuning daemon cancels the CPU binding core of the thread according to the relationship between the osd process and the thread number recorded in the memory.
[0072] For example: for the scene of 48 threads of CPU configuration, 10 osds of single node deployment, the specific implementation of the application is as follows:
[0073] osd_op_tp thread CPU set: osd_op_tp thread is the main business processing thread of ceph-osd, and there are 5 osd_op_tp threads in a single osd by default, and there are 50 osd_op_tp threads in 10 osds of a single storage node. Therefore, 2 / 3 of the CPU can be allocated to this thread, and the configuration item is osd_op_tp_cpuset="0-15,24-39". When the empty snapshot is recycled, 50 osd_op_tp threads of a single node execute in high concurrency, and only occupy 2 / 3 of the CPU, which does not affect the preemption of the remaining 1 / 3 of the CPU by other threads.
[0074] msgr-worker thread CPU set and ms_dispatch thread CPU set:
[0075] The msgr-worker and ms_dispatch threads are responsible for receiving, sending and internal distribution of network messages, have high coupling degree and busy business; therefore, 1 / 8 of the CPU can be allocated to the two types of threads, and the configuration items are msgr-worker_cpuset="16-18, 40-42" and ms_dispatch_cpuset="16-18, 40-42".
[0076] The CPU set of the osd_srv_heartbt thread:
[0077] In order to avoid the scenario of high concurrent preemption of CPU by osd_op_tp during empty snapshot recovery, the CPU of the osd_srv_heartbt heartbeat thread and the osd_op_tp thread needs to be isolated, and osd_srv_heartbt_cpuset="19-20, 43-44".
[0078] The CPU set of the tp_osd_cmd thread:
[0079] In order to avoid the scenario of high concurrent preemption of CPU by osd_op_tp during empty snapshot recovery, the CPU of the osd_srv_heartbt heartbeat thread and the osd_op_tp thread needs to be isolated, and osd_srv_heartbt_cpuset="19-20, 43-44".
[0080] Other threads and operating system: for the ceph-osd other threads, operating system and other business which are not bound to CPU cores, 1 / 12 of the remaining CPU resources can be used, and the above allocated CPU resources can also be preempted.
[0081] Embodiment 2
[0082] The application provides a Ceph performance optimization device, which comprises a classification module, an osd process timing detection module, an adding module, an osd core thread number acquisition module, a thread CPU set binding module,
[0083] The classification module divides the threads of the Ceph-osd into business processing threads, network message receiving and sending threads, message distribution threads, heartbeat threads, command interaction threads and other threads by analyzing the Ceph-osd system source code, the key business thread model and the runtime monitoring condition,
[0084] The newly added module adds a performance tuning daemon process, the sd process regularly detects the starting and exiting of the osd process of the storage node, the osd core thread number acquisition module acquires the thread number of the Ceph-osd thread through the htop tool, and the thread binding CPU set module binds the Ceph-osd thread to a specified CPU set according to the configuration item configuration through the taskset tool, so as to avoid thread resource preemption.
[0085] The information interaction, readable program process execution and the like among the modules in the above device are based on the same concept as the method embodiments of the present application, and the specific content can be referred to the description in the method embodiments of the present application, which will not be repeated here.
[0086] Similarly, the device of the present application adds a performance tuning daemon process and a configuration file, does not intrude into the ceph-osd program, belongs to an external hanging timing extraction mechanism, can bind threads of different categories to corresponding CPU sets when the ceph-osd runs, optimizes the binding strategy between threads and CPU cores, solves the problem of ceph-osd down caused by the large consumption of CPU during ceph-osd snaptrim, improves the reliability and stability of the entire ceph cluster, and improves the performance of the Ceph storage system.
[0087] It should be noted that not all steps and modules in the above processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above embodiments can be a physical structure or a logical structure, that is, some modules can be implemented by the same physical entity, or some modules can be implemented by multiple physical entities, or some modules can be implemented by some components in multiple independent devices.
[0088] The above-described embodiments are only preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Any equivalent replacement or transformation made by those skilled in the art based on the present application is within the protection scope of the present application. The protection scope of the present application is subject to the claims.
Claims
1. A Ceph performance tuning method, characterized by include: Step 1: By analyzing the Ceph-osd system source code, key business thread model and runtime monitoring, Ceph-osd threads are divided into business processing threads, network message sending and receiving threads, message distribution threads, heartbeat threads, command interaction threads and other threads. Step 2: Add a performance tuning daemon to monitor the startup and exit of the storage node osd process in real time. Use the htop tool to obtain the thread number of the Ceph-osd thread. Use the taskset tool to bind the Ceph-osd thread to the specified CPU set according to the configuration item to prevent the thread from preempting resources.
2. A Ceph performance tuning method according to claim 1, characterized in that The business processing thread in the Ceph-osd thread involved in step 1 is responsible for all key business processing, including data reading and writing, and data recovery. The network message sending and receiving threads involved are responsible for Ceph-osd's network communication, receiving external network data, and sending network data to the other end. The message distribution thread involved is responsible for converting the network data received by the network message sending and receiving thread into Ceph internal messages and forwarding them to the business processing thread for processing. The heartbeat thread involved is responsible for sending heartbeat detection messages to peerosd. The OSD command interaction thread involved is responsible for processing external instructions sent to OSD for configuration changes.
3. A Ceph performance tuning method according to claim 1, characterized in that Step 2 specifically includes: Step 21: Use the performance tuning daemon to monitor the startup and exit of the osd process in real time: Add notification scripts when the ceph-osd service is started and exited to inform the performance daemon ceph-osd process of changes, set the CPU set and cancel the CPU binding core through taskset, Step 22: Get the thread number from the performance tuning daemon: When the ceph-osd process is newly started, the performance tuning daemon obtains the core thread number of the ceph-osd service according to the htop command and records it. According to the thread names of the five major types of Ceph-osd threads, the thread number is obtained through the htop tool. Step 23: Use taskset to bind the CPU set of the thread through the performance tuning daemon: To avoid CPU resource preemption between threads, the five major types of threads are bound to different CPU cores, which are determined by the following five configuration items: osd_op_tp_cpuset configuration item, msgr-worker_cpuset configuration item, ms_dispatch_cpuset configuration item, osd_srv_heartbt_cpuset configuration item, and tp_osd_cmd_cpuset configuration item; The CPU set of the thread is dynamically set through the taskset command through the performance daemon, specifically through taskset -cCPU set -p thread number. Step 24: When the ceph-osd process exits, clean up the thread-bound cores through the performance tuning daemon: When the ceph-osd process exits, the performance tuning daemon cancels the CPU binding of the thread based on the relationship between the osd process and the thread number recorded in the memory.
4. A Ceph performance tuning method according to claim 1, characterized in that In step 2, for other ceph-osd threads, operating systems, and other services that are not bound to CPU cores, it is recommended to use the remaining preset share of CPU resources or preempt the above-allocated CPU resources.
5. A Ceph performance tuning device, characterized by Including classification module, osd process timing detection module, new module, osd core thread number acquisition module, thread binding CPU collection module, The classification module divides Ceph-osd threads into business processing threads, network message sending and receiving threads, message distribution threads, heartbeat threads, command interaction threads and other threads by analyzing the Ceph-osd system source code, key business thread models and runtime monitoring conditions. New modules have been added, including a performance tuning daemon. The sd process timing detection module monitors the startup and exit of the osd process of the storage node in real time. The osd core thread number acquisition module obtains the thread number of the Ceph-osd thread through the htop tool. The thread binding CPU set module uses the taskset tool to bind the Ceph-osd thread to the specified CPU set according to the configuration item configuration to prevent threads from preempting resources.
6. A Ceph performance tuning device according to claim 5, characterized in that The business processing thread in the Ceph-osd thread involved in the classification module is responsible for all key business processing, including data reading and writing, data recovery, The network message sending and receiving threads involved are responsible for Ceph-osd's network communication, receiving external network data, and sending network data to the other end. The message distribution thread involved is responsible for converting the network data received by the network message sending and receiving thread into Ceph internal messages and forwarding them to the business processing thread for processing. The heartbeat thread involved is responsible for sending heartbeat detection messages to peerosd. The OSD command interaction thread involved is responsible for processing external instructions sent to OSD for configuration changes.
7. A Ceph performance tuning device according to claim 5, characterized in that: The sd process timing detection module monitors the startup and exit of the osd process in real time through the performance tuning daemon process: Add notification scripts when the ceph-osd service is started and exited to inform the performance daemon ceph-osd process of changes, set the CPU set and cancel the CPU binding core through taskset, The osd core thread number acquisition module obtains the thread number through the performance tuning daemon: When the ceph-osd process is newly started, the performance tuning daemon obtains the core thread number of the ceph-osd service according to the htop command and records it. According to the thread names of the five major types of Ceph-osd threads, the thread number is obtained through the htop tool. The thread-binding CPU set module uses taskset to bind the CPU set of threads through the performance tuning daemon: To avoid CPU resource preemption between threads, the five major types of threads are bound to different CPU cores, which are determined by the following five configuration items: osd_op_tp_cpuset configuration item, msgr-worker_cpuset configuration item, ms_dispatch_cpuset configuration item, osd_srv_heartbt_cpuset configuration item, and tp_osd_cmd_cpuset configuration item; The CPU set of the thread is dynamically set through the taskset command through the performance daemon, specifically through taskset -cCPU set -p thread number. When the ceph-osd process exits, the thread-bound CPU set module cleans up the thread-bound cores through the performance tuning daemon: When the ceph-osd process exits, the performance tuning daemon cancels the CPU binding of the thread based on the relationship between the osd process and the thread number recorded in the memory.
8. A Ceph performance tuning device according to claim 5, characterized in that the thread The CPU set module recommends that other ceph-osd threads, operating systems, and other businesses that are not bound to CPU cores use the remaining preset share of CPU resources, or preempt the above-allocated CPU resources.