Optimization method for replacing metadata cache disk in distributed storage

Abstract

The invention discloses an optimization method for replacing a metadata cache disk of a distributed storage, which comprises the following steps of: 1, positioning a faulty metadata cache disk when the metadata cache disk alarms to replace the disk; 2, obtaining metadata information on the faulty metadata cache disk and stopping osd service; 3, metadata information on the fault metadata cache disk is returned to the corresponding HDD data disk based on the osd of which the service is stopped; 4, judging whether the new NVMe cache disk is in place or not, if yes, executing the step 5, and otherwise, starting the osd service; and 5, migrating the metadata migrated to the HDD data disk to a new NVMe cache disk again. According to the invention, the problem that the data disk needs to be remade when the metadata disk is replaced, so that a large amount of data is migrated and the service is influenced can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data cache, and in particular relates to an optimization method for replacing a metadata cache disk in distributed storage. Background Art

[0002] With the advent of the era of cloud computing and big data, distributed storage systems are gradually replacing traditional storage systems with excellent storage performance and low construction costs. Distributed storage systems generally store data in different nodes, and use redundancy strategies to ensure data security. When data errors occur, fault-tolerant mechanisms are used to restore invalid data. When data is updated, data is stored in different physical server nodes through redundancy strategies. Each physical server node has multiple HDD disks and 1-2 NVMe solid-state disks.

[0003] In order to improve the overall read and write performance of the distributed system, the metadata of the disk OSD is generally placed on NVMe, and the data is placed on HDD. Usually, one NVMe can serve as a metadata cache disk for 6-8 HDDs. As the server running time increases, the disk read and write time increases, and the probability of disk failure also increases. Distributed storage systems can generally replace disks that are about to fail in advance by monitoring the smart-related information of the disk. Currently, when replacing an NVMe disk that serves as a metadata cache disk, generally one NVMe will serve as a metadata cache disk for 6-8 HDD disks at the same time. When it is found that the NVMe disk is about to fail, the metadata cache disk needs to be destroyed when replacing the NVMe, which will affect the related 6-8 HDD data disks. After running for 1-2 years, these 6-8 HDD data disks will have a large amount of data on them. When these disks are reconstructed due to metadata redo, the amount of data will also be very large. Due to the dispersion of distributed storage data, the data reconstruction traffic will seriously affect the normal read and write business of the cluster. All HDD disks whose metadata are affected must be reconstructed, and data reconstruction cannot be performed on a single HDD disk one by one to reduce the impact of reconstruction on normal business. Summary of the invention

[0004] In view of the deficiencies of the above-mentioned prior art, the present invention provides an optimization method for replacing metadata cache disks in distributed storage, which reduces the large-scale reconstruction of data disks and avoids the impact on normal read and write services. The method can be performed OSD by OSD, further reducing the impact on the business and solving the problems existing in traditional distributed storage cache disk replacement.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] An optimization method for replacing a metadata cache disk in distributed storage includes the following steps:

[0007] Step 1. When the metadata cache disk alarm is triggered and the disk is replaced, locate the faulty metadata cache disk:

[0008] Step 2. Get the metadata information on the failed metadata cache disk and stop the osd service;

[0009] Step 3: Based on the OSD that has stopped service, migrate the metadata information on the failed metadata cache disk to the corresponding HDD data disk;

[0010] Step 4: Determine whether the new NVMe cache disk is in place. If so, proceed to step 5. Otherwise, start the osd service.

[0011] Step 5. Migrate the metadata migrated to the HDD data disk to the new NVMe cache disk again.

[0012] To optimize the above technical solutions, the specific measures taken also include:

[0013] The above step 1 determines the target cache disk to be replaced according to the monitoring alarm information and locates the faulty metadata cache disk.

[0014] The above-mentioned method of determining the target cache disk to be replaced and locating the faulty metadata cache disk based on the monitoring alarm information includes: finding the NVMe disk whose life is about to expire in the monitoring alarm platform through the distributed storage management platform, and locating the target NVMe metadata cache disk for replacement through the disk lighting function.

[0015] The above-mentioned step 2 includes: obtaining metadata information of multiple data disk osds corresponding to the cache disk; setting the distributed storage to maintenance mode, suspending data reconstruction during disk replacement; stopping the corresponding HDD data disk osd services one by one, and preparing for data migration.

[0016] The above step 3 includes: selecting an OSD that has stopped service to obtain its metadata LVM-related data information, and migrating the corresponding metadata db / wal back to the HDD data disk of the OSD; repeating the above process until all metadata on the cache disk are migrated to the corresponding HDD data disk.

[0017] The above step 5 includes: after the new NVMe cache disk is in place, the metadata db / wal migrated to the HDD data disk is gradually migrated to the new NVMe again, the relevant osd services are started, and the cluster maintenance mode is canceled.

[0018] A metadata-related lvm space is created on the new NVMe cache disk, and the metadata db / wal migrated to the HDD data disk is migrated to the lvm space again.

[0019] The present invention has the following beneficial effects:

[0020] The present invention does not need to reconstruct the data disk when the metadata cache disk is replaced. The metadata on the NVMe cache disk is temporarily migrated back to the data disk of the HDD for transfer. After the new cache disk NVMe is replaced, the metadata is migrated from the HDD to the new NVMe. This method reduces a large number of reconstructions of the data disk and avoids the impact on normal read and write services. In addition, the metadata can be replaced OSD by OSD, further reducing the impact on the business.

[0021] When the present invention migrates the metadata information on the failed metadata cache disk back to the corresponding HDD data disk, an OSD that has stopped service is selected to obtain its metadata LVM-related data information, and the corresponding metadata db / wal is migrated back to the HDD data disk of the OSD. This method can restore the metadata after migration, reduces data reconstruction, and can solve the problem that the data disk also needs to be redone when the metadata disk is replaced, thereby generating a large amount of data migration that affects the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The optimization method flow of replacing the metadata cache disk for the distributed storage of the present invention;

[0023] Figure 2 A schematic diagram of the drive letter information corresponding to the osd query of the present invention;

[0024] Figure 3 This is a schematic diagram of querying related information through a drive letter in the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0027] like Figure 1 As shown, the present invention proposes an optimization method for distributed cache disk replacement, comprising the following steps:

[0028] Step 1. Locate the faulty metadata cache disk: The monitoring and alarm platform determines the target cache disk to be replaced through SMART information, locates the disk through the storage management platform, and locks the replaced cache disk.

[0029] In the embodiment, the management platform of the distributed storage is opened, and the NVMe disk whose life is about to expire is found in the monitoring alarm platform, and the NVMe metadata cache disk that is the target of the disk replacement is located through the disk lighting function.

[0030] Step 2. Collect metadata information related to the old cache disk: obtain metadata information of multiple data disk osds corresponding to the cache disk; set the distributed storage to maintenance mode, suspend data reconstruction during disk replacement; stop the corresponding HDD data disk OSD services one by one, and prepare for data migration. The details are as follows:

[0031] (1) Query the corresponding drive letter information of osd, such as Figure 2 As shown:

[0032] cephdevicels-by-daemonosd.0

[0033] (2) Query relevant information through the drive letter, such as Figure 3 As shown:

[0034] ceph-volumelvmlist / dev / sdb

[0035] (3) Set the cluster maintenance mode to prevent data migration during operation

[0036] cephosdsetnoout

[0037] (4) Stop the corresponding osd service

[0038] systemctlstopceph-osd@0

[0039] (5) Repeat operations (1)-(4) until all osds corresponding to the metadata on the cache disk are completed.

[0040] Step 3. Migrate the metadata on the old cache disk back to the corresponding data disk HDD: Select an OSD that has stopped service to obtain its metadata LVM-related data information, and migrate the corresponding metadata db / wal back to the HDD of the OSD; repeat the above process until all metadata on the cache disk are migrated to the corresponding HDD data disk.

[0041] In the embodiment, a migrate script is developed and executed as follows:

[0042] migrate--osd-id0--osd-fsid49c18d27-021c-463c-8674-b0f39133f3f1--fromdb--targetceph- 2ebc35be-8eb6-451f-81c5-ccbc00aaef0f / osd-block-49c18d27-021c-463c-8674-b0f39133f3f1

[0043] Among them, osd-id is the osd number for metadata recommendation, osd-fsid is the cluster file system number of the osd service, from is the specified migration source, and --target is the lvm number of the osd HDD data disk. After the execution is completed, the metadata of the cache disk on the old disk is migrated back to the relevant data disk HDD.

[0044] Based on the above scheme, it can be known that when the present invention migrates the metadata information on the faulty metadata cache disk back to the corresponding HDD data disk, it selects an osd that has stopped service to obtain its metadata lvm-related data information, and migrates the corresponding metadata db / wal back to the HDD data disk of the osd. This method can restore the metadata after migration, reducing data reconstruction.

[0045] Step 4. Restart the previous osd service based on whether the new NVMe cache disk is in place.

[0046] If the NVMe disk is not in place, execute OSD to start the service:

[0047] systemctl startceph-osd@0

[0048] If the new NVMe is already in place, go to step 5.

[0049] Step 5. Replace the new NVMe cache disk: gradually migrate the metadata db / wal that was migrated to the HDD in the third step to the new NVMe again; start the relevant osd services; cancel the cluster maintenance mode. Specifically include:

[0050] (1) Create metadata-related LVM space on the new NVMe cache disk

[0051] new-db--osd-id0--osd-fsideed4359e-6a06-476a-aa21-5401f1a7fa0d--targetceph-3ce e9f59-8353-4337-b374-1743f7d4073a / osd-db-688a6755-6e01-4303-994a-6f1158c8a6bc

[0052] Among them, osd-id is the osd number of the metadata to be migrated, osd-fsid is the cluster file system number of the osd service, and target is the metadata lvm volume information of the NVMe cache disk to be migrated.

[0053] (2) Migrate metadata from the HDD disk to the metadata lvm volume created above

[0054] migrate--osd-id0--osd-fsideed4359e-6a06-476a-aa21-5401f1a7fa0d--fromdata--targetce ph-3cee9f59-8353-4337-b374-1743f7d4073a / osd-db-688a6755-6e01-4303-994a-6f1158c8a6bc

[0055] Among them, osd-id is the number of the osd for metadata recommendation, osd-fsid is the number of the cluster file system served by the osd, from is the specified migration source, and --target is the lvm number of the osd metadata cache disk. After the execution is completed, the metadata is migrated back to the new cache disk.

[0056] (3) Restart the relevant OSD services to complete the cache disk operation.

[0057] In summary, the present invention does not need to reconstruct the data disk when the metadata cache disk is replaced. The metadata on the NVMe cache disk is temporarily migrated back to the data disk of the HDD for transfer. After the new cache disk NVMe is replaced, the metadata is migrated from the HDD to the new NVMe. This method reduces a large number of reconstructions of the data disk and avoids the impact on normal read and write services. In addition, the replacement of metadata can be performed OSD by OSD, further reducing the impact on the business. The present invention can solve the problem that the data disk also needs to be redone when the metadata disk is replaced, thereby generating a large amount of data migration that affects the business.

[0058] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0059] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. An optimization method for replacing a metadata cache disk in distributed storage, characterized in that: The steps include: Step 1. When the metadata cache disk alarm is triggered and the disk is replaced, locate the faulty metadata cache disk: Step 2. Get the metadata information on the failed metadata cache disk and stop the osd service; Step 3: Based on the OSD that has stopped service, migrate the metadata information on the failed metadata cache disk to the corresponding HDD data disk; Step 4: Determine whether the new NVMe cache disk is in place. If so, proceed to step 5. Otherwise, start the osd service. Step 5. Migrate the metadata migrated to the HDD data disk to the new NVMe cache disk again.

2. According to claim 1, a distributed storage metadata cache disk optimization method is characterized in that: The step 1 determines the target cache disk to be replaced according to the monitoring alarm information and locates the faulty metadata cache disk.

3. The optimization method for replacing a metadata cache disk in distributed storage according to claim 2, characterized in that: The method of determining the target cache disk to be replaced and locating the faulty metadata cache disk according to the monitoring alarm information includes: finding the NVMe disk whose life is about to expire in the monitoring alarm platform through the distributed storage management platform, and locating the target NVMe metadata cache disk for replacement through the disk lighting function.

4. The optimization method for replacing a metadata cache disk in distributed storage according to claim 1, characterized in that: The step 2 includes: obtaining metadata information of multiple data disk osds corresponding to the cache disk; setting the distributed storage to maintenance mode and suspending data reconstruction during disk replacement; stopping the corresponding HDD data disk osd services one by one to prepare for data migration.

5. The optimization method for replacing a metadata cache disk in distributed storage according to claim 1, characterized in that: The step 3 includes: selecting an OSD that has stopped service to obtain its metadata LVM-related data information, and migrating the corresponding metadata db / wal back to the HDD data disk of the OSD; repeating the above process until all metadata on the cache disk are migrated to the corresponding HDD data disk.

6. The optimization method for replacing a metadata cache disk in distributed storage according to claim 1, characterized in that: The step 5 includes: after the new NVMe cache disk is in place, the metadata db / wal migrated to the HDD data disk is gradually migrated to the new NVMe again, the relevant osd service is started, and the cluster maintenance mode is canceled.

7. The optimization method for replacing a metadata cache disk in distributed storage according to claim 6, characterized in that: A metadata-related lvm space is created on the new NVMe cache disk, and the metadata db / wal migrated to the HDD data disk is migrated to the lvm space again.

Images