Method for quickly recovering ceph node

Abstract

The invention relates to a method for quickly recovering ceph nodes, which comprises the following steps of: when a fault node exists in a system, moving an OSD (On Screen Display) data disk of the fault node to a recovery node; and if the ceph osd disk is normal, checking the network configuration and disk configuration of the recovery node ceph osd disk to ensure that the network configuration and disk configuration of the recovery node disk are consistent with those of the fault node. A ceph basic package is installed, a ceph configuration file is configured, and a ceph osd block soft link is configured; recovering the configuration file under the osd path directory, and installing the reconfigured ceph osd disk to a fault node of the ceph distributed storage system; and starting the osd service, and waiting for the ceph cluster state to recover to normal. According to the method for quickly recovering the data through non-triggering time-consuming data reconstruction, the data recovery efficiency is improved by more than 10 times, the node recovery time is shortened, and the risk of data loss in the recovery process is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed storage systems, and in particular relates to a method for quickly recovering a ceph node. Background Art

[0002] Ceph is a unified distributed storage system consisting of multiple nodes. As the cluster size grows, the number of nodes increases, and the probability of node failure increases. When a node in the ceph distributed storage system fails, the process of handling the failed node in the prior art is as follows: The above solution has the following problems: the data recovery time of the cluster is linearly related to the amount of data carried by the failed node. The larger the data volume, the longer the data recovery time; there is a risk of data loss during cluster data recovery; the cluster data recovery process will affect the read and write performance of the cluster; the recovery process is complicated and inconvenient for operation and maintenance. In the scenario where the node is abnormal due to a system disk or operating system failure, the ceph OSD disk on the node is normal, and data recovery through the above recovery method is unnecessary.

[0003] Therefore, how to improve the fault recovery process when a faulty node occurs in the prior art, achieve rapid recovery of node failures in the Ceph distributed storage system, and reduce the risk of data loss is a technical problem that urgently needs to be solved. Summary of the invention

[0004] The object of the present invention is to provide a method for quickly recovering a ceph node, so as to quickly recover a node failure in a ceph distributed storage system.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows: A method for quickly recovering a ceph node includes the following steps: S1: collect the specified data of each node in the ceph distributed storage system in real time, and analyze the specified data of each node collected in real time to determine whether there is a faulty node in each node. If so, execute step S2; S2: Move the OSD data disk of the failed node to the recovery node; S3: Check whether the ceph osd disk is normal. If so, check the network configuration and disk configuration of the ceph osd disk of the recovery node to determine whether the network configuration and disk configuration of the recovery node disk are consistent with those of the faulty node. If so, execute step S4; if the cephosd disk is abnormal, directly replace the ceph osd disk of the faulty node in the ceph distributed storage system; S4: Install the ceph basic package, configure the ceph configuration file, and configure the ceph osd block soft link; S5: Restore the configuration files in the osd path directory and install the reconfigured ceph osd disk to the failed node of the ceph distributed storage system; S6: Start the osd service and wait for the ceph cluster status to return to normal.

[0006] Preferably, the network configuration of the ceph osd disk of the recovery node in step S3 includes public network configuration, cluster network configuration, message transmission type configuration, firewall configuration and MTU setting; The disk configuration includes storage engine selection, core configuration parameters, permission settings, configuration files, and data placement strategies.

[0007] Preferably, the ceph basic package in step S4 includes an rpm data package and a python data package required by the ceph cluster; The ceph configuration file includes ceph cluster global configuration information, monitor, mgr configuration information, osd global configuration information and recovery node osd configuration information.

[0008] Preferably, the specific process of configuring the ceph osd block soft link is as follows: According to the osd corresponding to the ceph osd disk partition of the current node, create a soft link between the osdblock file and the corresponding disk partition path, and modify the user and user group to which the soft link file belongs to be the ceph service running user.

[0009] Preferably, the specific process of restoring the configuration file in the osd path directory in step S5 is as follows: According to the cluster and current osd information, restore the bluefs, ceph_fsid, fsid, kv_backend, magic, mkfs_done, ready, require_osd_release, type, and whoami configuration files in the osd path directory.

[0010] Preferably, the specific process of starting the osd service in step S6 is as follows: Start the osd service of the current node through the systemctl start ceph-osd@{osd_id} command.

[0011] Preferably, at the same time as step S6, the ceph cluster status is monitored in real time during the specified period, and it is determined whether the ceph cluster status returns to normal during the specified period. If not, the backup cluster recovery mechanism is started: S61: Execute ceph osd out {osd_id} on the ceph monitor node to set all osds on the failed node to out, triggering the data recovery mechanism; S62: Execute ceph osd crush remove {osd_name} to remove all osds on the faulty node from the crushmap, triggering the uniform distribution of data among all nodes; S63: Execute ceph osd rm {osd_id} to delete all osds on the faulty node; S64: Install the ceph basic package on the new node: including the rpm package and python package required by the ceph cluster; S65: Configure a ceph configuration file on the new node, wherein the ceph configuration file includes ceph cluster global configuration information, monitor configuration information, mgr configuration information, and new node osd configuration information; S66: Execute the ceph-osd --mkfs command on the new node to initialize the osd of the current node; S67: Execute the ceph osd crush command on the new node to add the osd of the current node to the crushmap; S68: Execute the systemctl start ceph-osd@{osd_id} command on the new node to start the osd service of the current node; S69: Wait for the Ceph cluster status to be normal.

[0012] The beneficial effects of the present invention include: The method for quickly recovering a ceph node provided by the present invention, when a faulty node is detected in the system through a node fault monitoring process, the data on the ceph osd disk of the faulty node is moved to a redundant node, and the ceph osd disk on the faulty node is removed from the ceph distributed storage system; if the cephosd disk is normal, if so, the network configuration and disk configuration of the ceph osd disk of the recovery node are checked to ensure that the network configuration and disk configuration of the recovery node disk are consistent with those of the faulty node. Install the ceph basic package, configure the ceph configuration file, and configure the ceph osd block soft link; restore the configuration file under the osd path directory, and install the reconfigured cephosd disk to the faulty node of the ceph distributed storage system; start the osd service, and wait for the ceph cluster status to return to normal. By not triggering time-consuming data reconstruction and performing a method for rapid data recovery, the data recovery efficiency is improved by more than 10 times, the node recovery time is shortened, and the risk of data loss during the recovery process is reduced.

[0013] First, when there is a faulty node in the ceph distributed storage system, and the ceph osd disk of the faulty node is normal, it means that the node is unavailable due to an operating system failure. Then, remove the disk of the faulty node, ensure that the network configuration and disk configuration of the recovery node are consistent with those of the faulty node, install the ceph basic package, configure the ceph configuration file, and configure the ceph osd block soft link. Restore the configuration file in the osd path directory, start the osd service, and wait for the ceph cluster status to return to normal to perform fault recovery. This process effectively improves data recovery efficiency by not triggering time-consuming data reconstruction and performing data recovery quickly. In addition, the method of restoring nodes by ensuring that the network configuration and disk configuration of the recovery node are the same as those of the faulty node shortens the node recovery time and reduces the risk of data loss during the recovery process.

[0014] Secondly, when there is a faulty node in the ceph distributed storage system and the ceph osd disk of the faulty node is abnormal, the fault can be recovered by directly replacing the ceph osd disk to avoid the long time and risk of data loss when the entire system uses the cluster data recovery mechanism to restore the cluster.

[0015] Again, when there is a faulty node in the Ceph distributed storage system and the disk itself is normal, but due to various reasons, the system fault recovery cannot be achieved by removing the faulty node disk for fault recovery, the data recovery mechanism of the Ceph cluster is executed, and the data recovery mechanism of the Ceph cluster is used as the final alternative fault recovery mechanism to ensure that the entire Ceph set can achieve fault recovery when a fault occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a schematic diagram of the architecture of the ceph distributed storage system of the present invention.

[0017] Figure 2 The present invention is a flowchart of a method for quickly recovering a ceph node.

[0018] Figure 3 It is a flow chart of the standby cluster recovery mechanism of the present invention. DETAILED DESCRIPTION

[0019] The following is combined with Figure 1~Figure 3 The present invention is further described in detail: Example 1 See attached Figure 1-2 As shown, a method for quickly recovering a ceph node includes the following steps: S1: collect the specified data of each node in the ceph distributed storage system in real time, analyze the specified data of each node collected in real time, and determine whether there is a faulty node in each node. If so, execute step S2. Real-time monitoring of the ceph distributed storage system is achieved by real-time collection and analysis of the specified data of each node in the ceph distributed storage system.

[0020] S2: Move the OSD data disk of the failed node to the recovery node. By moving the OSD data disk of the failed node to the recovery node, the entire ceph distributed storage system can be ensured to operate normally, which can effectively reduce the risk of data loss in the system, and then remove the ceph osd disk of the failed node. By removing the disk, the failure recovery of the disk reconfiguration is performed separately, without the need to execute the data recovery mechanism of the ceph cluster in the system, which greatly reduces the time of system failure recovery.

[0021] S3: Check whether the ceph osd disk is normal. If so, check the network configuration and disk configuration of the ceph osd disk of the recovery node to determine whether the network configuration and disk configuration of the recovery node disk are consistent with those of the faulty node. If so, execute step S4; if the cephosd disk is abnormal, directly replace the ceph osd disk of the faulty node in the ceph distributed storage system. When the disk itself is damaged, fault recovery can only be performed by replacing the disk; S4: Install the ceph basic package, configure the ceph configuration file, and configure the ceph osd block soft link; S5: Restore the configuration files in the osd path directory and install the reconfigured ceph osd disk to the failed node of the ceph distributed storage system; S6: Start the osd service and wait for the ceph cluster status to return to normal.

[0022] In the above-mentioned method of removing the disk in this embodiment for fault recovery, it is necessary to ensure that the ceph osd disk itself is normal, otherwise it is directly replaced. When executing this solution, it is necessary to ensure that the new node operating system has been installed; the cephosd disk of the failed node needs to be inserted into the new node, and the network configuration of the new node needs to be consistent with the failed node. When a ceph node fails, the method of quickly recovering data can be performed without triggering time-consuming data reconstruction, and the data recovery efficiency is improved by more than 10 times. The method of restoring a node by ensuring that the network configuration and disk configuration of the recovery node are the same as those of the failed node shortens the node recovery time and reduces the risk of data loss during the recovery process.

[0023] Preferably, the network configuration of the ceph osd disk of the recovery node in step S3 includes public network configuration, cluster network configuration, message transmission type configuration, firewall configuration and MTU setting; The disk configuration includes storage engine selection, core configuration parameters, permission settings, configuration files, and data placement strategies.

[0024] Preferably, the ceph basic package in step S4 includes an rpm data package and a python data package required by the ceph cluster; The ceph configuration file includes ceph cluster global configuration information, monitor, mgr configuration information, osd global configuration information and recovery node osd configuration information.

[0025] Example 2 Based on Example 1, the specific process of configuring the ceph osd block soft link is as follows: According to the osd corresponding to the ceph osd disk partition of the current node, create a soft link between the osdblock file and the corresponding disk partition path, and modify the user and user group to which the soft link file belongs to be the ceph service running user.

[0026] In this embodiment, the specific process of restoring the configuration file in the osd path directory in step S5 is as follows: According to the cluster and current osd information, restore the bluefs, ceph_fsid, fsid, kv_backend, magic, mkfs_done, ready, require_osd_release, type and whoami configuration files in the osd path directory. The specific process of starting the osd service in step S6 is as follows: Start the osd service of the current node through the systemctl start ceph-osd@{osd_id} command.

[0027] Example 3 Based on Example 1 or Example 2, see Figure 3 At the same time as step S6, the ceph cluster status is monitored in real time during the specified period, and it is determined whether the ceph cluster status has returned to normal during the specified period. If not, the backup cluster recovery mechanism is started: S61: Execute ceph osd out {osd_id} on the ceph monitor node to set all osds on the failed node to out, triggering the data recovery mechanism. This can ensure data integrity and consistency, while also minimizing the risk of data loss. The recovery process can automatically repair data damage or loss to ensure data reliability.

[0028] S62: Execute ceph osd crush remove {osd_name} to remove all osds on the failed node from the crushmap, triggering the uniform distribution of data among the nodes. This is mainly to ensure the uniform distribution of data among the nodes after the osd is removed, so as to improve the performance and reliability of the cluster.

[0029] S63: Execute ceph osd rm {osd_id} to delete all osds on the faulty node; S64: Install the ceph basic package on the new node: including the rpm package and python package required by the ceph cluster; S65: Configure a ceph configuration file on the new node, wherein the ceph configuration file includes ceph cluster global configuration information, monitor configuration information, mgr configuration information, and new node osd configuration information; S66: Execute the ceph-osd --mkfs command on the new node to initialize the osd of the current node; S67: Execute the ceph osd crush command on the new node to add the osd of the current node to the crushmap; S68: Execute the systemctl start ceph-osd@{osd_id} command on the new node to start the osd service of the current node; S69: Wait for the Ceph cluster status to be normal.

[0030] The above cluster recovery mechanism is used as a backup ceph distributed storage system failure recovery mechanism. When failure recovery cannot be achieved by removing the disk for reconfiguration, it can be used as the final failure recovery option to ultimately achieve system failure recovery.

[0031] In summary, the method for quickly recovering ceph nodes provided by the present invention, when a faulty node is detected in the system through a node fault monitoring process, the data on the ceph osd disk of the faulty node is moved to a redundant node, and the ceph osd disk on the faulty node is removed from the ceph distributed storage system; if the cephosd disk is normal, if so, the network configuration and disk configuration of the ceph osd disk of the recovery node are checked to ensure that the network configuration and disk configuration of the recovery node disk are consistent with the faulty node. Install the ceph basic package, configure the ceph configuration file, and configure the ceph osdblock soft link; restore the configuration file under the osd path directory, and install the reconfigured cephosd disk to the faulty node of the ceph distributed storage system; start the osd service, and wait for the ceph cluster status to return to normal. By not triggering time-consuming data reconstruction and performing a method for rapid data recovery, the data recovery efficiency is improved by more than 10 times, shortening the node recovery time and reducing the risk of data loss during the recovery process.

[0032] When there is a faulty node in the ceph distributed storage system, and the ceph osd disk of the faulty node is normal, it means that the node is unavailable due to an operating system failure. Then, the disk of the faulty node is removed to ensure that the network configuration and disk configuration of the recovery node are consistent with those of the faulty node, install the ceph basic package, configure the ceph configuration file, and configure the ceph osd block soft link. Restore the configuration file in the osd path directory, start the osd service, and wait for the ceph cluster status to return to normal to perform fault recovery. This process effectively improves data recovery efficiency by not triggering time-consuming data reconstruction and performing data recovery quickly. The method of restoring nodes by ensuring that the network configuration and disk configuration of the recovery node are the same as those of the faulty node shortens the node recovery time and reduces the risk of data loss during the recovery process.

[0033] When there is a faulty node in the ceph distributed storage system and the ceph osd disk of the faulty node is not normal, the ceph osd disk is directly replaced for fault recovery, avoiding the long time and data loss risk of the entire system recovering the cluster through the cluster data recovery mechanism. When there is a faulty node in the ceph distributed storage system and the disk itself is normal, but due to various reasons, the method of removing the faulty node disk for fault recovery still cannot achieve system fault recovery, then execute the ceph cluster data recovery mechanism, and use the ceph cluster data recovery mechanism as the final alternative fault recovery mechanism to ensure that the entire ceph set can achieve fault recovery when a fault occurs.

Claims

1. A method for quickly recovering a ceph node, characterized in that: The following steps are involved: S1: collect the specified data of each node in the ceph distributed storage system in real time, and analyze the specified data of each node collected in real time to determine whether there is a faulty node in each node. If so, execute step S2; S2: Move the OSD data disk of the failed node to the recovery node; S3: Check whether the ceph osd disk is normal. If so, check the network configuration and disk configuration of the ceph osd disk of the recovery node to determine whether the network configuration and disk configuration of the recovery node disk are consistent with those of the faulty node. If so, execute step S4; if the ceph osd disk is abnormal, directly replace the ceph osd disk of the faulty node in the ceph distributed storage system; S4: Install the ceph basic package, configure the ceph configuration file, and configure the ceph osd block soft link; S5: Restore the configuration files in the osd path directory and install the reconfigured ceph osd disk to the failed node of the ceph distributed storage system; S6: Start the osd service and wait for the ceph cluster status to return to normal.

2. A method for quickly recovering a ceph node according to claim 1, characterized in that: The network configuration of the ceph osd disk of the restored node in step S3 includes public network configuration, cluster network configuration, message transmission type configuration, firewall configuration and MTU setting; The disk configuration includes storage engine selection, core configuration parameters, permission settings, configuration files, and data placement strategies.

3. A method for quickly recovering a ceph node according to claim 1, characterized in that: The ceph basic package in step S4 includes the rpm data package and python data package required by the ceph cluster; The ceph configuration file includes ceph cluster global configuration information, monitor, mgr configuration information, osd global configuration information and recovery node osd configuration information.

4. A method for quickly recovering a ceph node according to claim 3, characterized in that: The specific process of configuring the cephosd block soft link is as follows: According to the osd corresponding to the ceph osd disk partition of the current node, create a soft link between the osd block file and the corresponding disk partition path, and modify the user and user group to which the soft link file belongs to be the ceph service running user.

5. A method for quickly recovering a ceph node according to claim 1, characterized in that: The specific process of restoring the configuration file in the osd path directory in step S5 is as follows: According to the cluster and current osd information, restore the bluefs, ceph_fsid, fsid, kv_backend, magic, mkfs_done, ready, require_osd_release, type, and whoami configuration files in the osd path directory.

6. A method for quickly recovering a ceph node according to claim 1, characterized in that: The specific process of starting the osd service in step S6 is as follows: Start the osd service of the current node through the systemctl start ceph-osd@{osd_id} command.

7. A method for quickly recovering a ceph node according to claim 1, characterized in that: At the same time as step S6, the ceph cluster status is monitored in real time during the specified period, and it is determined whether the ceph cluster status has returned to normal during the specified period. If not, the backup cluster recovery mechanism is started: S61: Execute ceph osd out {osd_id} on the ceph monitor node to set all osds on the failed node to out, triggering the data recovery mechanism; S62: Execute ceph osd crush remove {osd_name} to remove all osds on the faulty node from the crushmap, triggering the uniform distribution of data among all nodes; S63: Execute ceph osd rm {osd_id} to delete all osds on the faulty node; S64: Install the ceph basic package on the new node: including the rpm package and python package required by the ceph cluster; S65: Configure a ceph configuration file on the new node, wherein the ceph configuration file includes ceph cluster global configuration information, monitor configuration information, mgr configuration information, and new node osd configuration information; S66: Execute the ceph-osd --mkfs command on the new node to initialize the osd of the current node; S67: Execute the ceph osd crush command on the new node to add the osd of the current node to the crushmap; S68: Execute the systemctl start ceph-osd@{osd_id} command on the new node to start the osd service of the current node; S69: Wait for the Ceph cluster status to be normal.

Images