Ceph distributed storage cluster fault cutoff optimization method, electronic equipment and storage medium

Abstract

The invention provides a Ceph distributed storage cluster fault cutoff optimization method, electronic equipment and a storage medium, and relates to the field of cloud computing. The method comprises the following steps: deploying a Ceph distributed storage cluster, recording a mapping relation between each node and a corresponding OSD, and generating a comparison table; deploying a Zookeeper service at each node of the cluster, and creating a persistent root node; configuring a session heartbeat timeout parameter of the Zookeeper, and when the server does not receive a heartbeat packet of the client within a set time, triggering a session timeout event; the client monitors the state change of a child node under the root node, and when the node is detected to be offline, a Zookeeper callback function is called; marking all OSDs corresponding to the fault nodes as down states according to the comparison table, and enabling the collating groups to quickly enter Peering states; and synchronously updating the comparison table when the capacity of the cluster nodes is expanded or reduced or the OSD is increased or decreased. Compared with Ceph heartbeat detection, the fault cutoff time realized by the method can be shortened by at least more than 50%.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to a Ceph distributed storage cluster fault interruption optimization method, electronic equipment, and storage medium. Background Art

[0002] In the Ceph distributed storage system, fault detection, such as node power failure and network failure, mainly relies on the heartbeat mechanism between OSDs and MON / OSDs. By quickly marking the OSD of the failed node and updating the OSDMap, the client is notified that it will no longer access the OSD of the node. However, because Ceph itself prevents cluster network jitter and heartbeat storms (each disk in the cluster is an OSD, and OSDs send heartbeats to each other), Ceph will experience IO interruptions of nearly minutes in the event of failures such as node power failure and network interruption. This will cause IO congestion in upper-layer applications and business interruption (usually tens of seconds to minutes). Summary of the Invention

[0003] Purpose of the invention: To propose a Ceph distributed storage cluster fault interruption optimization method, and to propose an electronic device and storage medium for storing this method, which can significantly shorten the Ceph interruption time caused by failure, thereby reducing the impact of the failure on the continuity of upper-layer business.

[0004] The first aspect of the present invention provides a Ceph distributed storage cluster fault interruption optimization method, the steps are as follows:

[0005] Deploy the Ceph distributed storage cluster, record the mapping relationship between each node and the corresponding OSD, and generate the comparison table Cluster_OSD.map;

[0006] Deploy the Zookeeper service on each node in the cluster and create a persistent root node / server-ceph;

[0007] Configure the session heartbeat timeout parameters of the Zookeeper service. When the server does not receive the client's heartbeat packet within the set time, a session timeout event is triggered.

[0008] The client monitors the status changes of child nodes under the / server-ceph node. When it detects that a node is offline, it calls the Zookeeper callback function.

[0009] According to the comparison table Cluster_OSD.map, all OSDs corresponding to the failed node are marked as down, so that the placement group quickly enters the Peering state;

[0010] When the cluster nodes are expanded or reduced or the number of OSDs is increased or decreased, the comparison table Cluster_OSD.map is updated synchronously.

[0011] In a further embodiment of the first aspect, the step of recording the mapping relationship between each node and the corresponding OSD includes: obtaining the OSD list of each node through the cephosd ls-tree command, forming a comparison table of a key-value pair structure, where the key is the node name and the value is the OSD number set under the node.

[0012] In a further embodiment of the first aspect, the Zookeeper service is deployed in such a manner that one node serves as a leader and the remaining nodes serve as followers to form a Zookeeper cluster.

[0013] In a further embodiment of the first aspect, the session heartbeat timeout parameters of the Zookeeper service are configured as: tickTime=2000ms, sessionTimeout=3, that is, the server triggers a session timeout when it does not receive a heartbeat packet within 6 seconds.

[0014] In a further embodiment of the first aspect, the child node is a temporary node. When the node goes online, a temporary child node is created under / server-ceph. When the node goes offline, Zookeeper automatically removes the node.

[0015] In a further embodiment of the first aspect, the step of marking the OSD corresponding to the faulty node as down includes: executing cephosd down osd.$i through a command line, where i is the OSD number corresponding to the faulty node.

[0016] In a further embodiment of the first aspect, when a node comes online, a temporary child node is created under the / server-ceph node to indicate that the node service is started normally.

[0017] In a further embodiment of the first aspect, when the cluster nodes are expanded or reduced in capacity, the Zookeeper service of the corresponding storage nodes is synchronously deployed or deleted, and the comparison table Cluster_OSD.map is updated.

[0018] The Watcher mechanism of Zookeeper is used to monitor the creation, modification and deletion events of nodes, and triggers a callback function when it detects that a node is offline.

[0019] In a second aspect of the present invention, an electronic device is proposed, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the Ceph distributed storage cluster fault interruption optimization method as described in the first aspect is implemented.

[0020] In a third aspect of the present invention, a computer-readable storage medium is proposed, which stores at least one executable instruction. When the executable instruction is run on an electronic device, the electronic device executes the Ceph distributed storage cluster fault interruption optimization method as described in the first aspect.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) Significantly shorten the outage time and improve the reliability of distributed storage clusters: Through the Ceph fault outage optimization based on ZooKeeper, the service outage time of the distributed storage system in abnormal situations such as node power failure and network card failure can be significantly shortened, with the improvement effect exceeding 50%.

[0023] (2) Reduce the impact of business interruptions: By shortening the outage time through Zookeeper, business systems deployed on distributed storage clusters can recover faster from node and network card failures, minimizing the impact of hardware failures on upper-layer businesses. For businesses with high data real-time requirements, such as online transactions and real-time monitoring, this can ensure the continuous and stable operation of the business, avoid business interruptions or data loss caused by storage failures, and thus improve business reliability and stability.

[0024] (3) Rapid data access recovery: When a distributed storage cluster fails, application access to stored data is affected. Shortening the outage time allows applications to regain access to data more quickly, reducing business stagnation caused by data inaccessibility and enabling business operations to resume normal operations as quickly as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 FIG. 1 is a schematic diagram of setting the OSD to the dwon state in an embodiment.

[0026] Figure 2 This is the data flow diagram of Zookeeper heartbeat detection in the embodiment.

[0027] Figure 3 This is an overall flow chart of the Ceph distributed storage cluster failure interruption optimization method in the embodiment.

[0028] Figure 4 The following is a schematic diagram of three storage servers running in the cluster in the embodiment. DETAILED DESCRIPTION

[0029] In the following description, numerous specific details are provided to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without one or more of these details. In other instances, certain technical features well known in the art have not been described to avoid confusion with the present invention.

[0030] Before describing the embodiments, the professional terms that appear in the following text are first explained.

[0031] Node: Zookeeper's data model is a tree structure, with each node called a ZNode. A ZNode can contain data and child nodes, similar to the directory structure of a file system.

[0032] Session: A connection between a client and a Zookeeper server. A session has state, including temporary nodes, permissions, and more. A session has a timeout; if no heartbeat from the client is received within this time, the session expires.

[0033] Ephemeral Node: A special ZNode whose lifecycle is bound to the session that created it. If the session expires, the node will be automatically deleted.

[0034] Persistent Node: A ZNode whose lifecycle is not dependent on the session. Even if the client session that created it expires, the node still exists.

[0035] Watcher: An event listener. The client can register a Watcher on the Zookeeper node. When the node status changes, Zookeeper will notify the client that has set the Watcher.

[0036] Monitor (MON): Monitor maintains various charts that show the status of the cluster, including the monitor chart, OSD chart, placement group (PG) chart, and CRUSH chart. Monitor is the Ceph monitoring function responsible for maintaining the health of the entire cluster and providing consistent decision-making.

[0037] OSD (Object Storage Device): An OSD is a process responsible for responding to client requests and returning specific data. A Ceph cluster typically has multiple OSDs. Their primary functions include storing data, handling data replication, recovery, restocking, balancing data distribution, and providing relevant data to Ceph Monitors.

[0038] PG (Placement Groups): PG is a logical concept that contains multiple OSDs. The introduction of the PG layer is actually to better distribute and locate data.

[0039] In the Ceph distributed storage system, fault detection, such as node power failure and network failure, mainly relies on the heartbeat mechanism between OSDs and MON / OSDs. By quickly marking the OSD of the failed node and updating the OSDMap, the client is notified that it will no longer access the OSD of the node.

[0040] OSD heartbeat detection:

[0041] An OSD will establish a Heartbeat connection with three other OSDs:

[0042] (1) Between OSD nodes under the same PG, because OSD nodes belonging to the same PG will store copies of the same data, if a failure occurs, it will directly affect the availability of the data.

[0043] (2) The two adjacent nodes on the left and right of the OSD have a close physical connection with itself, for example, they may be connected to the same switch.

[0044] (3) If the number of peers establishing heartbeats is less than osd_heartbeat_min_peers, the OSD will continue to establish heartbeat connections with several OSDs that are closer to it.

[0045] MON heartbeat detection:

[0046] (1) To avoid heartbeat storms, each heartbeat detection needs to be performed at a certain interval (osd_heartbeat_interval). When the heartbeat detection finds that an OSD has not responded for a period of time (osd_heartbeat_grace), it is considered that the OSD has an abnormality and a message will be sent to the Monitor to report the failure of the OSD.

[0047] (2) When Monitor receives mon_osd_min_down_reporters OSD reports of the same OSD failure from different fault domains (mon_osd_reporter_subtree_level), it will put the OSD into the dwon state, see Figure 1 .

[0048] For example, when mon_osd_reporter_subtree_level = host and mon_osd_min_down_reporters = 2, the monitor sets osd.3 to down.

[0049] (3) In some extreme cases, the fault detection of OSD cannot meet the fault mechanism conditions of OSD heartbeat detection. In this case, the heartbeat detection of monitor is needed to provide a bottom line, but since the number of monitors is usually much smaller than the number of OSDs, the detection cycle of monitor will be longer. Figure 2 , OSD sends heartbeat information to monitor every osd_mon_report_interval seconds to inform monitor of its status. If monitor does not receive heartbeat message from an OSD after mon_osd_report_timeout time, it will mark the OSD as down.

[0050] When Ceph experiences node, network, or cabinet failures, read and write services at the upper layer may be interrupted for a period of time. This interruption occurs because it takes time for the monitor to detect that an OSD is down. During this time, the client cannot determine which OSD is down, and read and write requests sent to the failed OSD will be stuck. Only when the monitor detects an OSD failure and updates the OSDmap can the client obtain the updated OSDmap without being blocked.

[0051] Whether it's a network failure or a node failure, it will ultimately be reflected in the OSD status. Ceph's fault detection mechanism detects OSD failures at two levels: OSDs directly check each other through a heartbeat mechanism, and monitors check OSDs through a heartbeat mechanism.

[0052] In Zookeeper, when a node is powered off, that is, when the state of a node (ZNode) in Zookeeper changes, such as node creation, modification, deletion, etc., Zookeeper will notify all Watchers registered on the node. The overall process is as follows, and the logical flow chart is as follows Figure 3 .

[0053] Since Zookeeper's own heartbeat detection node modification has added multiple judgments, for example, the heartbeat is sent once every 2 seconds. If there is a problem after sending it three times, the node is considered offline, which greatly reduces the risk of heartbeat misjudgment. At the same time, since Zookeeper heartbeat packets are sent by different node processes, unlike the heartbeats sent between multiple OSDs, the number of heartbeat packets can be reduced, reducing the possibility of heartbeat storms.

[0054] The embodiment of the present invention adopts an innovative method to introduce Zookeeper to significantly shorten the long interruption time of Ceph faults. In Zookeeper, when a node is powered off, that is, when the status of a node (ZNode) in Zookeeper changes, such as the creation, modification, deletion of the node, Zookeeper will notify all Watchers registered on the node. After observing that the node fails and goes offline, all OSD Markdowns of the node are marked down. The overall process is shown in the following figure. Figure 3 , the specific steps are as follows:

[0055] Step 1. After the Ceph distributed storage cluster is deployed, record the nodes and all OSDs under each node. For example, node91, node92, and node93 form a distributed storage cluster. Use the cephosd ls-tree node91 command to record all OSDs of node node91. All nodes form a comparison table Cluster_OSD.map: The structure is as follows:

[0056] [node91]1 3 5......

[0057] [node92]0 2 4......

[0058] [node93]6 7 8......

[0059] Step 2: Deploy the Zookeeper service on each node in the cluster, with one node as the leader and the remaining nodes as followers.

[0060] Step 3: Create a persistent root node / server-ceph

[0061] Step 4. Set the session heartbeat super configuration of each node in the Zookeeper cluster. Configure in the zoo.cfg file, tickTime = 2000ms (configurable), sessionTimeout = 3 (configurable). That is, if the server does not receive the client's heartbeat packet within 2*3=6s, it will be considered that the client has been disconnected, triggering the Session timeout event.

[0062] Step 5. The client monitors the online / offline status of the child nodes under the / server-ceph node. When a node comes online, it indicates that the node service is started normally. When a node goes offline, it indicates that the node service is stopped due to a fault.

[0063] Step 6: When a service node failure is detected, the Zookeeper callback function is called. According to the node and OSD comparison table generated in step 1, all OSDs of the failed node are set to down, that is, cephosd down osd.$i

[0064] Step 7. When the node comes online, create a child node (non-persistent node) under the / server-ceph node. When the server fails and the connection is disconnected, ZK will automatically remove it.

[0065] Step 8. When the ceph cluster expands / contracts nodes or adds / deletes OSDs, the node and OSD comparison table Cluster_OSD.map recorded in step 1 is updated synchronously.

[0066] Step 9: When a node is expanded or reduced in the Ceph cluster, the Zookeeper service of the storage node is deployed and deleted simultaneously.

[0067] For example, after the client is started, it detects three storage servers running in the cluster. Figure 4 .

[0068] client start working..

[0069] [node91,node92,node93]

[0070] #After a failed server is powered off, the server list is updated and node92 is offline

[0071] [node91,node93]

[0072] #After node92 is down, the callback function is called to read the list of all OSDs corresponding to node92 and put all OSDs of the node down

[0073] cephosd down osd.$i

[0074] #After the faulty server is online, the server list is updated and node92 is online

[0075] [node91,node92,node93]

[0076] In this example, Zookeeper is used to quickly detect node failures in the Ceph cluster and mark down any Up OSDs associated with the failed node. This allows the PGs associated with the failed OSDs to quickly enter the Peering state, allowing them to more quickly transition to the Active+degraded state and accept client I / O. This can reduce the outage time by at least 50% compared to Ceph's own heartbeat detection, as shown in Table 1.

[0077] Table 1: Comparison of mon node before and after optimization

[0078]

[0079] This patent is applicable to multiple scenarios using Ceph distributed storage and has a wide range of applications. Typical application scenarios are as follows:

[0080] (1) Cloud storage: Applicable to cloud storage systems built based on Ceph, including public cloud and private cloud systems.

[0081] (2) Virtualization: In server virtualization scenarios, Ceph distributed storage clusters can serve as the storage backend for virtual machines, providing block storage services for them. For example, in an enterprise's virtualized data center, Ceph distributed storage clusters can provide backend storage services for a large number of virtual machines, improving server resource utilization.

[0082] (3) Container backend storage: In a container environment, Ceph serves as a persistent storage solution for containers. Data that applications in containers need to access is stored in the Ceph cluster, ensuring data persistence and reliability.

[0083] (4) Video storage and distribution: For video surveillance, Ceph serves as a backend solution for storing large amounts of video data. It supports real-time writing of video data generated by surveillance cameras for long-term storage. Ceph storage also supports fast retrieval and playback of video data, making it easy for users to view historical videos.

[0084] (5) Video on demand and live streaming platform: Ceph serves as the storage backend, providing users with a smooth video playback experience. Its high throughput and low latency performance can meet the needs of a large number of users accessing videos simultaneously, ensuring fast video loading and playback.

[0085] The logic behind the methods disclosed in the above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs.

[0086] When the computer instruction or computer program is loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part.The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website, a computer, a server or a data center to another website, a computer, a server or a data center by a wired (such as infrared, wireless, microwave, etc.) mode.The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center containing one or more available media sets.The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD) or a semiconductor medium.The semiconductor medium can be a solid-state drive.

[0087] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0090] While the present invention has been shown and described with reference to certain preferred embodiments, it is not to be construed as limiting the invention itself, and various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A Ceph distributed storage cluster fault interruption optimization method, characterized in that: The steps include: Deploy the Ceph distributed storage cluster, record the mapping relationship between each node and the corresponding OSD, and generate the comparison table Cluster_OSD.map; Deploy the Zookeeper service on each node in the cluster and create a persistent root node / server-ceph; Configure the session heartbeat timeout parameters of the Zookeeper service. When the server does not receive the client's heartbeat packet within the set time, a session timeout event is triggered. The client monitors the status changes of child nodes under the / server-ceph node. When it detects that a node is offline, it calls the Zookeeper callback function. According to the comparison table Cluster_OSD.map, all OSDs corresponding to the failed node are marked as down, so that the placement group quickly enters the Peering state; When the cluster nodes are expanded or reduced or the number of OSDs is increased or decreased, the comparison table Cluster_OSD.map is updated synchronously.

2. A Ceph distributed storage cluster fault interruption optimization method according to claim 1, characterized in that: The step of recording the mapping relationship between each node and the corresponding OSD includes: obtaining the OSD list of each node through the cephosd ls-tree command, forming a comparison table of a key-value pair structure, wherein the key is the node name and the value is the OSD number set under the node.

3. A Ceph distributed storage cluster fault interruption optimization method according to claim 1, characterized in that: The Zookeeper service is deployed in such a way that one node serves as a leader and the remaining nodes serve as followers to form a Zookeeper cluster.

4. A Ceph distributed storage cluster fault interruption optimization method according to claim 1, characterized in that: The session heartbeat timeout parameters of the Zookeeper service are configured as follows: tickTime=2000ms, sessionTimeout=3, that is, the session timeout is triggered when the server does not receive a heartbeat packet within 6 seconds.

5. A Ceph distributed storage cluster fault interruption optimization method according to claim 1, characterized in that: The child node is a temporary node. When the node goes online, a temporary child node is created under / server-ceph. When the node goes offline, Zookeeper automatically removes the node.

6. A Ceph distributed storage cluster fault interruption optimization method according to claim 1, characterized in that: The step of marking the OSD corresponding to the faulty node as being in a down state comprises: executing cephosd down osd.$i through a command line, where i is the OSD number corresponding to the faulty node.

7. A Ceph distributed storage cluster fault interruption optimization method according to claim 1, characterized in that: When the node comes online, a temporary child node is created under the / server-ceph node to indicate that the node service has started normally.

8. A Ceph distributed storage cluster fault interruption optimization method according to claim 1, characterized in that: When the cluster nodes are expanded or reduced, the Zookeeper service of the corresponding storage nodes is deployed or deleted synchronously, and the comparison table Cluster_OSD.map is updated. The Watcher mechanism of Zookeeper is used to monitor the creation, modification and deletion events of nodes, and triggers a callback function when it detects that a node is offline.

9. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the Ceph distributed storage cluster fault interruption optimization method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one executable instruction. When the executable instruction is executed on the electronic device, the electronic device executes the Ceph distributed storage cluster fault interruption optimization method according to any one of claims 1 to 8.

Images