A CEPH metadata migration and upgrade method and device based on distributed database

Abstract

The present application provides a CEPH metadata migration and upgrade method and device based on a distributed database, the method comprising: modifying the write mark of the migration bucket to double write, and updating the latest uploaded / deleted / modified metadata to SHRAD and the distributed database at the same time; migrating the existing metadata of SHRAD to the distributed database; performing a full comparison between SHRAD and the distributed database, and then performing two incremental comparisons, migrating the HEADER data of the migration bucket on SHARD to the distributed database, switching the read and write state of the migration bucket to a single-read single-write database, and turning on the mark-delete mark of the migration bucket; performing a full data comparison between SHRAD and the distributed database again, and deleting the data marked as mark-delete in the migration bucket from the distributed database; cleaning up the metadata of SHARD; the method migrates the metadata stored on SHARD in the original bucket to the distributed database without interrupting the user's business, realizing the function of unlimited number of object storage in a single bucket of the user, and improving the quality of user use.

Description

Technical Field

[0001] The present application relates to the technical field of CEPH metadata, and in particular to a CEPH metadata migration and upgrade method and device based on a distributed database. Background Art

[0002] In the field of distributed storage, CPEH distributed file system is currently the most widely used, and object storage has been widely used in the distributed field due to its advantages of low storage cost, high scalability, security, reliability, and strong flexibility. In object storage, object storage uses data and metadata separation to store files uploaded by users, where data is stored in the DATA pool and metadata is stored in the INDEX pool. In object storage, buckets are containers used by users to store object data, and objects uploaded by users must be stored in buckets. When creating a bucket, a fixed number of shards (SHARDs) will be created in the INDEX pool according to the configuration of the cluster to store the metadata of the object. If a bucket has too many SHARDs, it will cause the user to have a high latency when performing LIST operations on objects in the bucket; if a bucket has too few SHARDs, it will cause too many metadata entries to be stored in a single SHARD, which will affect the performance of user uploads and downloads. At the same time, since the number of SHARDs is fixed when creating a bucket, and there is also an upper limit on the metadata entry data that each SHARD can store, there is also an upper limit on the number of objects that can be stored in a single bucket, which cannot meet the user's needs to store a large number of objects in a single bucket.

[0003] At this stage, there is an urgent need to develop a CEPH metadata migration and upgrade method and device based on a distributed database to solve the above-mentioned technical problems. Summary of the invention

[0004] The present application shows a CEPH metadata migration and upgrade method and device based on a distributed database.

[0005] In a first aspect, the present application shows a CEPH metadata migration and upgrade method based on a distributed database, the method comprising:

[0006] If the distributed database is available, change the write flag of the migration bucket to double write, and update the latest uploaded / deleted / modified metadata to SHRAD and the distributed database at the same time;

[0007] Migrate SHRAD's existing metadata to a distributed database;

[0008] Compare the full data of SHRAD and distributed databases, and record the first moment T1;

[0009] Compare the incremental data in SHRAD and the distributed database from the first time T1 to the second time T2, and migrate the HEADER data of the migration bucket on SHRAD to the distributed database at the same time;

[0010] Compare the incremental data in SHRAD and the distributed database after the second time T2, and migrate the HEADER data of the migration bucket on SHRAD to the distributed database at the same time, switch the read-write state of the migration bucket to a single-read single-write database, and enable the mark-delete flag of the migration bucket;

[0011] Compare the full data between SHRAD and the distributed database again, and delete the data marked for deletion in the migration bucket from the distributed database.

[0012] Clean up the metadata in the SHARD INDEX pool.

[0013] According to the method of the first aspect of the present application, the latest uploaded / deleted / modified metadata is updated to SHRAD and the distributed database simultaneously, specifically including:

[0014] First write the latest uploaded / deleted / modified metadata to SHRAD;

[0015] If writing to SHRAD fails, it returns failure directly without writing to the distributed database.

[0016] If the write to SHRAD is successful, the latest uploaded / deleted / modified metadata will continue to be written to the distributed database;

[0017] If writing to the distributed database fails, a new structure is added to record the object that failed to be written to the database, and success is returned.

[0018] According to the method of the first aspect of the present application, the stock metadata of SHRAD is migrated to the distributed database, specifically: all SHARDs in the migration bucket are traversed, and the original metadata of the current object in each SHARD is directly migrated to the distributed database.

[0019] According to the method of the first aspect of the present application, a full data comparison is performed on SHRAD and the distributed database, specifically including:

[0020] Record the current time node as the first moment T1;

[0021] Traverse the metadata of all objects on the SHARD in the migration bucket and check whether the corresponding metadata exists in the distributed database;

[0022] If the corresponding metadata does not exist in the distributed database, the metadata of the object on SHARD is directly copied to the distributed database;

[0023] If the corresponding metadata exists in the distributed database, compare the mtime in the metadata of the object on the SHARD with the mtime in the corresponding metadata on the distributed database to see if they are consistent;

[0024] If the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten to the distributed database.

[0025] According to the method of the first aspect of the present application, the incremental data in the SHRAD and the distributed database during the period from the first time T1 to the second time T2 are compared, and the HEADER data of the migration bucket on the SHRAD is migrated to the distributed database at the same time, specifically including:

[0026] Record the current time node as the second moment T2;

[0027] Obtain the metadata of the object on SHARD, and determine whether mtine in the metadata of the object on SHARD is greater than the first moment T1;

[0028] If mtine is greater than the first moment T1, check whether the corresponding metadata exists in the distributed database;

[0029] If the corresponding metadata does not exist in the distributed database, the metadata of the object on the SHARD is directly copied to the distributed database; if the corresponding metadata exists in the distributed database, the mtime in the metadata of the object on the SHARD is compared with the mtime in the corresponding metadata on the distributed database to see if they are consistent; if the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten in the distributed database;

[0030] Traverse all SHARDs in the migration bucket;

[0031] Obtain all metadata whose mtine is greater than the first moment T1 from the distributed database, and check whether the corresponding metadata exists in SHARD;

[0032] If the corresponding metadata does not exist in SHARD, the corresponding metadata in the distributed database will be deleted; if the corresponding metadata exists in SHARD, the metadata of the object on SHARD will be migrated to the distributed database, and finally the HEADER data of the migration bucket on SHARD will be migrated to the distributed database.

[0033] According to the method of the first aspect of the present application, the incremental data in the SHRAD and the distributed database after the second time T2 are compared, and the HEADER data of the migration bucket on the SHRAD is migrated to the distributed database at the same time, the read-write state of the migration bucket is switched to a single-read single-write database, and the deletion mark of the migration bucket is turned on, which specifically includes:

[0034] Compare the incremental data in SHRAD and the distributed database after the second time T2;

[0035] After the incremental comparison is completed, lock the migration bucket;

[0036] Switch the read / write state of the migration bucket to a single-read single-write database, and record the current switching time as the third moment T3;

[0037] Migrate the HEADER data of the migration bucket on SHARD to the distributed database;

[0038] Enable the delete flag of the migration bucket;

[0039] Unlock the migration bucket.

[0040] According to the method of the first aspect of the present application, the full data of SHRAD and the distributed database are compared again, and the data marked as marked for deletion in the migration bucket is deleted from the distributed database, specifically including:

[0041] Get the metadata of the object on SHARD and check whether the corresponding metadata exists in the distributed database;

[0042] If the corresponding metadata does not exist in the distributed database, the metadata of the object on SHARD is directly copied to the distributed database;

[0043] If the corresponding metadata exists in the distributed database, all SHARDs in the migration bucket are traversed;

[0044] Get metadata from the distributed database and check whether the corresponding metadata exists on SHARD;

[0045] If the corresponding metadata exists in SHARD, the data marked as deleted in the migration bucket on SHARD is deleted from the distributed database;

[0046] If the corresponding metadata does not exist in SHARD, the corresponding metadata whose mtime is less than the third time T3 is deleted from the distributed database.

[0047] In a second aspect, the present application shows a CEPH metadata migration and upgrade device based on a distributed database, the device comprising:

[0048] The first processing module is configured to, when the distributed database is available, modify the write flag of the migration bucket to double write, and update the latest uploaded / deleted / modified metadata to SHRAD and the distributed database at the same time;

[0049] A second processing module is configured to migrate the stock metadata of SHRAD to a distributed database;

[0050] The third processing module is configured to compare the full amount of data between SHRAD and the distributed database, and record the first time T1;

[0051] The fourth processing module is configured to compare the incremental data in the SHRAD and the distributed database during the period from the first time T1 to the second time T2, and simultaneously migrate the HEADER data of the migration bucket on the SHRAD to the distributed database;

[0052] The fifth processing module is configured to compare the incremental data in the SHRAD and the distributed database after the second time T2, and simultaneously migrate the HEADER data of the migration bucket on the SHRAD to the distributed database, switch the read-write state of the migration bucket to a single-read single-write database, and turn on the mark-delete flag of the migration bucket;

[0053] The sixth processing module is configured to perform full data comparison between the SHRAD and the distributed database again, and delete the data marked as marked for deletion in the migration bucket from the distributed database;

[0054] The seventh processing module is configured to clean up metadata in the INDEX pool of SHARD.

[0055] In a third aspect, the present application shows an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute the CEPH metadata migration and upgrade method based on a distributed database as described in any of the above aspects.

[0056] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the CEPH metadata migration and upgrade method based on a distributed database as described in any of the above aspects.

[0057] In a fifth aspect, the present application illustrates a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the CEPH metadata migration and upgrade method based on a distributed database as described in any of the above aspects.

[0058] The technical solution provided by this application may have the following beneficial effects:

[0059] 1. Migrate metadata in the bucket of object storage from SHARD to the distributed database, so that a single bucket of object storage can store unlimited objects and reduce the latency of metadata-related operations;

[0060] 2. Use error logs to record objects that failed to operate the database during the dual-write phase, so as to facilitate the subsequent tracking of these objects. At the same time, ensure that there are no failure records in the database before switching to a single-read single-write database to ensure database availability;

[0061] 3. A multi-round comparison scheme ensures data consistency between SHARD and distributed databases before and after migration without interrupting user services. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a step flow of a CEPH metadata migration and upgrade method based on a distributed database in this application Figure 1 .

[0063] Figure 2 It is a schematic diagram of data reading and writing on SHARD in the prior art.

[0064] Figure 3 This is a schematic diagram of step 101 in the CEPH metadata migration and upgrade method based on a distributed database of the present application.

[0065] Figure 4 This is a schematic diagram of step 102 in the CEPH metadata migration and upgrade method based on a distributed database of the present application.

[0066] Figure 5 This is a flowchart of step 103 in the CEPH metadata migration and upgrade method based on a distributed database of the present application.

[0067] Figure 6 This is a flowchart of step 104 in the CEPH metadata migration and upgrade method based on a distributed database of the present application.

[0068] Figure 7 This is a flow chart of step 105 in the CEPH metadata migration and upgrade method based on a distributed database of the present application.

[0069] Figure 8 This is a flow chart of querying and modifying metadata processing when locking a bucket in the CEPH metadata migration and upgrade method based on a distributed database of the present application.

[0070] Fig. 9 This is a flow chart of step 106 in the CEPH metadata migration and upgrade method based on a distributed database of the present application.

[0071] Fig.10 The steps of the CEPH metadata migration and upgrade method based on the distributed database of this application are as follows: Figure 2 .

[0072] Fig.11 It is a structural block diagram of a CEPH metadata migration and upgrade device based on a distributed database of the present application.

[0073] Fig.12 It is a block diagram of an electronic device of the present application.

[0074] Fig.13 It is a block diagram of a computer storage medium of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] Glossary:

[0077] SAHRD Sharding: A single bucket of CEPH object storage is divided into different SHARDs for storing object metadata.

[0078] CEPH: A distributed file system with high reliability and scalability.

[0079] Distributed storage: A data storage technology that uses the disks of each machine in the enterprise through the network and distributes these storage resources to form a virtual storage device, distributing the data to each machine.

[0080] INDEX pool: dedicated to storing object metadata.

[0081] In view of the current object storage metadata storage method in CEPH, which results in the number of objects stored in a single bucket cannot be too many, the LIST performance of objects in the bucket is poor, and it needs to be compatible with all object data in the original bucket and cannot interrupt user business during the upgrade process. A CEPH metadata migration and upgrade solution based on a distributed database is designed. This solution can migrate the metadata stored on SHARD in the user's original bucket to the distributed database without interrupting user business, and use the characteristics of the distributed database to achieve the function of unlimited number of objects stored in a single bucket of the user, and use the orderly storage of data in the database to greatly improve the performance of background thread execution such as LIST and LC in the user bucket, thereby improving the quality of user use.

[0082] The following is an explanation of the CEPH metadata migration and upgrade method based on a distributed database.

[0083] Reference Figure 1 , shows a flowchart of the steps of a CEPH metadata migration and upgrade method based on a distributed database of the present application, the method can be applied to an electronic device, wherein the method can specifically include the following steps:

[0084] Step 101: If the distributed database is available, change the write flag of the migration bucket to double write, and update the latest uploaded / deleted / modified metadata to SHRAD and the distributed database at the same time;

[0085] Step 102: Migrate the SHRAD metadata to the distributed database;

[0086] Step 103: perform a full data comparison between SHRAD and the distributed database, and record the first moment T1;

[0087] Step 104: Compare the incremental data in the SHRAD and the distributed database during the period from the first time T1 to the second time T2, and simultaneously migrate the HEADER data of the migration bucket on the SHRAD to the distributed database;

[0088] Step 105: Compare the incremental data in SHRAD and the distributed database after the second time T2, and migrate the HEADER data of the migration bucket on SHRAD to the distributed database at the same time, switch the read-write state of the migration bucket to a single-read single-write database, and enable the mark-delete flag of the migration bucket;

[0089] Step 106: perform a full data comparison between SHRAD and the distributed database again, and delete the data marked as marked for deletion in the migration bucket from the distributed database;

[0090] Step 107: Clean up the metadata in the INDEX pool of the SHARD.

[0091] It should be noted that the newly added data (i.e., incremental data) in this embodiment adds the following fields to the original RGWBucketInfo structure:

[0092]

[0093] Among them: bucket_index_read_flag indicates whether the bucket reads data from SHARD or distributed database; bucket_index_write_flag indicates the location where the bucket writes data; bucket_index_log_flag indicates the location where the bucket consumes BILOG; is_marked_deleted indicates whether the bucket has marked deletion enabled.

[0094] The values ​​of bucket_index_read_flag and bucket_index_log_flag are: 0 or 1, where: 0 means reading metadata or consuming BILOG from SHARD, and 1 means reading metadata or consuming BILOG from a distributed database. The values ​​of bucket_index_write_flag are: 0, 1, 2, where: 0 and 1 mean the same as the above bucket_index_read_flag and bucket_index_log_flag, and 2 means writing metadata to both SHARD and the distributed database. The value of is_marked_deleted is 0 or 1, where 0 means not enabling bucket mark deletion, and 1 means enabling bucket mark deletion.

[0095] Because BILOG in CEPH is used to record the operation of users modifying metadata, users generally perform cross-domain replication or data synchronization in different clusters such as multiple sites, which leads to a certain time difference between consuming BILOG and generating BILOG. This may cause a part of BILOG to not be consumed on SHARD after switching to a single-read single-write database. If BILOG consumption is switched to a distributed database while switching to a single-read single-write database, the unconsumed BILOG remaining on SHARD will be lost, resulting in missing replicated data. Therefore, bucket_index_log_flag is needed to indicate whether BILOG is consumed from SHARD or a distributed database.

[0096] The metadata in the original SHARD is migrated to the distributed database. The metadata in SHARD can be roughly divided into three categories: 1. HEADER statistical data; 2. INDEX metadata corresponding to each object; 3. BILOG record operation log data. At the same time, in order to ensure that the distributed database can store the metadata of all buckets in a cluster, the data in the distributed database is designed to be in the form of {KEY, VALUE1, VALUE2..}, where key is globally unique and is used to save object name and bucket name, and VALUES saves the relevant metadata parameters in the original SHARD.

[0097] The newly added data storage metadata are as follows:

[0098] 1. The BUCKET_ID of each bucket in the cluster is globally unique, so the database storage format of HEADER statistical data is {{BUCKET_ID}.header, VALUES}. 2. The INDEX metadata corresponding to each object is stored in the distributed database in the format of {{BUCKET_ID}_{OBJECT_KEY}, VALUES}. 3. BILOG needs to record the SHARD ID on the original SHARD, so the record format of BILOG in the database is {{BUCKET_ID}.{SHARD_ID}, VALUES}. Different KEY formats are used to distinguish statistical data, object metadata, and BILOG log data.

[0099] Because the newly deployed distributed database may be unstable, a new structure is used to record objects that fail to be written to the database during the dual-write phase. The specific structure is as follows:

[0100]

[0101] Among them, bucket_name indicates the BUCKET that failed to be written to the database, shard_id indicates the specific SHARD that was successfully written to the SHARD but failed to be written to the database, obj indicates the object that failed to be written, the string in the map indicates the specific object name, and int indicates the operation that needs to be performed, such as deletion, addition, update, etc.

[0102] The CEPH metadata migration and upgrade method based on a distributed database of the present application adopts the method of migrating the metadata of the objects stored in the SHARD to the distributed database, and improves the performance of the metadata LIST by taking advantage of the fact that the data stored in the distributed database is in the lexicographic order of the primary key. In addition, the number of data entries stored in the database itself is theoretically unlimited. Therefore, after the distributed database is used to store the object metadata, there will be no upper limit on the number of objects that can be stored in a single bucket. At the same time, since the SHARD is essentially a RADOS object, the metadata of the object is stored in the form of KEY-VALUE, and all the metadata of the object is stored in the VALUE, and the distributed database can use multiple columns to store the object metadata, the metadata of the object in the VALUE is split into multiple columns, and the object name in the first column is used as the primary key. This can greatly improve the speed of the background application of the object, such as GC, LC, cross-domain replication, etc., and bring a better user experience.

[0103] In step 101, when the distributed database is available, the write flag of the migration bucket is changed to double write, and the latest uploaded / deleted / modified metadata is updated to SHRAD and the distributed database at the same time.

[0104] In some embodiments, step 101 of the CEPH metadata migration and upgrade method based on a distributed database of the present application specifically includes:

[0105] First write the latest uploaded / deleted / modified metadata to SHRAD;

[0106] If writing to SHRAD fails, it returns failure directly without writing to the distributed database.

[0107] If the write to SHRAD is successful, the latest uploaded / deleted / modified metadata will continue to be written to the distributed database;

[0108] If writing to the distributed database fails, a new structure is added to record the object that failed to be written to the database, and success is returned.

[0109] Specifically, Figure 2 The figure shows the original process of metadata reading / modification / uploading on SHARD. Figure 3 It shows the process of updating the latest uploaded / deleted / modified metadata to SHRAD and the distributed database at the same time after the double write mark is turned on in the method of the present application.

[0110] like Figure 3 As shown in the figure, if the dual write flag is enabled for the migration bucket, the SHARD and the distributed database need to be written at the same time during the metadata writing phase, that is, write to the SHARD first, then write to the distributed database. During this process, if writing to the SHARD fails, a failure is directly returned, and writing to the distributed database is not continued; if writing to the SHARD succeeds but writing to the distributed database fails, the above rgw_db_write_failed_obj structure is added to record the object that failed to write to the database, and success is returned to continue the subsequent process.

[0111] Since transactions need to be guaranteed during metadata modification, and SHARD and distributed databases do not have transaction conditions, there may be inconsistencies between the modified data written to the distributed database and SHARD. For example, when uploading multi-version objects, modifying multi-version OLH metadata and multi-version version numbers, this metadata needs to read the original data and then modify it. At this time, the metadata in SHARD is read first, then modified and written back to SHARD, and then the metadata is read from the distributed database and written back to the database. However, at this time, there may be inconsistencies between the metadata read from SHARD and the distributed database, which will cause incorrect data to be written in the distributed database. This problem is corrected by the subsequent comparison process of the data in the distributed database. During the entire double-write phase, only the INDEX metadata corresponding to each object is double-written, while the HEADER data and BILOG data are not double-written, because the HEADER data does not exist in the database before migration, and the modification of the HEADER data needs to be read first, then modified and written back, so it is not meaningful to double-write the HEADER data; and BILOG data is a log data, which only needs to be recorded on one side.

[0112] In step 102, the stock metadata of SHRAD is migrated to the distributed database.

[0113] In some embodiments, step 102 in the CEPH metadata migration and upgrade method based on a distributed database of the present application is specifically: traverse all SHARDs in the migration bucket, and directly migrate the original metadata of the current object in each SHARD to the distributed database.

[0114] For details, see Figure 4 As shown, in this embodiment, the migration of existing metadata is to migrate the object-related metadata on the original SHARD to the distributed database. During the migration process, all SHARDs in the migration bucket are traversed, and the metadata of the current object in each SHARD is directly migrated to the distributed database.

[0115] It is worth noting that during the migration process, the migration bucket still has user services, that is, there are continuous metadata write and delete operations, and the metadata on SHARD is constantly updated. Therefore, during the migration process, only most of the data in SHARD and the distributed database can be guaranteed to be consistent, and strong consistency cannot be guaranteed. However, the migration process can distribute the migration tasks to different gateways, and use the performance of different gateways to improve the efficiency of the migration and minimize the time spent on the migration.

[0116] In step 103, a full data comparison is performed between SHRAD and the distributed database, and the first time T1 is recorded.

[0117] In some embodiments, step 103 in the CEPH metadata migration and upgrade method based on a distributed database of the present application specifically includes:

[0118] Record the current time node as the first moment T1;

[0119] Traverse the metadata of all objects on the SHARD in the migration bucket and check whether the corresponding metadata exists in the distributed database;

[0120] If the corresponding metadata does not exist in the distributed database, the metadata of the object on SHARD is directly copied to the distributed database;

[0121] If the corresponding metadata exists in the distributed database, compare the mtime in the metadata of the object on the SHARD with the mtime in the corresponding metadata on the distributed database to see if they are consistent;

[0122] If the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten to the distributed database.

[0123] For details, see Figure 5 As shown, the specific process of full comparison in this embodiment is as follows: first, record the current time node, that is, the first moment T1, which is used for the next incremental comparison to avoid the need to compare all data for each comparison. First, traverse the metadata on all SHARDs of the migration bucket to check whether the metadata of the object exists in the distributed database. If the metadata does not exist, directly insert the metadata into the distributed database; if the metadata exists, it is necessary to compare the mtime in the metadata in the distributed database. If the mtime on both sides is inconsistent, it means that there is data inconsistency between the distributed database and the SHARD. At this time, the SHARD is used as the standard, and the metadata data on the SHARD needs to be directly overwritten into the distributed database. After traversing all SHARDs of the migration bucket, it is necessary to read the metadata of the migration bucket from the distributed database and compare it with the metadata on the SHARD. If there is a certain data in the distributed database, but not on the SHARD, it may be a distributed database error write, such as the case of the distributed database error write in the multi-version case mentioned above. At this time, it is necessary to delete the records in the distributed database to ensure that the data on the distributed database and the SHARD are consistent.

[0124] In step 104, the incremental data in the SHRAD and the distributed database during the period from the first time T1 to the second time T2 are compared, and at the same time, the HEADER data of the migration bucket on the SHRAD is migrated to the distributed database.

[0125] In some embodiments, step 104 in the CEPH metadata migration and upgrade method based on a distributed database of the present application specifically includes:

[0126] Record the current time node as the second moment T2;

[0127] Obtain the metadata of the object on SHARD, and determine whether mtine in the metadata of the object on SHARD is greater than the first moment T1;

[0128] If mtine is greater than the first moment T1, check whether the corresponding metadata exists in the distributed database;

[0129] If the corresponding metadata does not exist in the distributed database, the metadata of the object on the SHARD is directly copied to the distributed database; if the corresponding metadata exists in the distributed database, the mtime in the metadata of the object on the SHARD is compared with the mtime in the corresponding metadata on the distributed database to see if they are consistent; if the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten in the distributed database;

[0130] Traverse all SHARDs in the migration bucket;

[0131] Obtain all metadata whose mtine is greater than the first moment T1 from the distributed database, and check whether the corresponding metadata exists in SHARD;

[0132] If the corresponding metadata does not exist in SHARD, the corresponding metadata in the distributed database will be deleted; if the corresponding metadata exists in SHARD, the metadata of the object on SHARD will be migrated to the distributed database, and finally the HEADER data of the migration bucket on SHARD will be migrated to the distributed database.

[0133] For details, see Figure 6As shown in the figure, in this embodiment, the process of incremental comparison and migrating the HEADER is as follows: Since a full comparison was performed in step 103, the first moment T1 was recorded at this time. Before T1, the data had been compared once, and at this time, the data in the SHARD and the distributed database was consistent. Therefore, when performing the comparison this time, the data in this part can be skipped. Therefore, when performing the SHARD comparison, the mtime in the metadata can be compared with T1. If mtime < T1, it means that the modification time of the object metadata is before T1, and the time when the object was uploaded or modified is before T1, and the object metadata can be skipped. When filtering the object metadata in the distributed database, mtime can be selected as the filter option to filter out the metadata whose upload or modification time is after T1. If there are records in these metadata that do not exist on the SHARD, they will be deleted. After the incremental comparison is completed, the HEADER metadata in the SHARD is migrated to the database, and it is checked whether the HEADER migration is normal.

[0134] In step 105, the incremental data in the SHRAD and the distributed database after the second moment T2 is compared, and at the same time, the HEADER data of the migration bucket on the SHARD is migrated to the distributed database, and the read-write state of the migration bucket is switched to a single-read and single-write database, and the marking deletion flag of the migration bucket is enabled.

[0135] In some embodiments, step 105 in the method for migrating and upgrading CEPH metadata based on a distributed database of the present application specifically includes:

[0136] Compare the incremental data in the SHRAD and the distributed database after the second moment T2;

[0137] After the incremental comparison is completed, lock the migration bucket;

[0138] Switch the read-write state of the migration bucket to a single-read and single-write database, and record the current switching time as the third moment T3;

[0139] Migrate the HEADER data of the migration bucket on the SHARD to the distributed database;

[0140] Enable the marking deletion flag of the migration bucket;

[0141] Unlock the migration bucket.

[0142] Specifically, refer to Figure 7As shown, in this embodiment, the process of switching the read-write state, migrating the HEADER, and enabling the mark-deletion is as follows: first, a round of incremental comparison is performed to ensure that the data on the SHARD from T2 to the current time node is consistent with the data in the distributed database. After the incremental comparison is completed, the migration bucket needs to be locked to migrate the HEADER data in the SHARD. It is worth noting that before locking, it is necessary to wait for the currently entered request to be completed to ensure that there is no metadata modification during the process of switching the migration bucket state and migrating the HEADER. When the migration bucket is locked, it is necessary to modify the read-write state of the migration bucket to a single-read single-write database, and record the current time as the third moment T3; then migrate the HEADER data, and at the same time enable the mark-deletion mark of the migration bucket. After the mark-deletion is enabled, deleting the object in the migration bucket after T3 will only delete the object's data, but will not actually delete the object's metadata, but will mark the object's metadata with a mark-deletion mark, and the metadata with this mark cannot be LIST or GET, that is, it is actually invisible to the user. The standard deletion method is used to ensure that the data in the distributed database is not deleted after the switch, so as to avoid the last round of comparison to migrate the data to the distributed database again, which may cause problems for user services. Finally, the migration bucket is unlocked and the front-end business of the migration bucket is restored.

[0143] See also Figure 8 As shown, in this embodiment, the user request processing is specifically as follows: during the lock bucket process, after the migration bucket is locked, if the metadata in the migration bucket is queried, it can be returned normally; but if the metadata in the migration bucket needs to be modified, such as adding metadata, deleting metadata, modifying certain attributes of metadata, etc., it needs to return failure. At the same time, since the time of migrating the HEADER is very short, in order to reduce the impact on the user, a retry is performed internally. If the migration bucket has been unlocked during the retry process, the user request will still return success, but if the migration bucket has not been unlocked after the maximum number of retries is reached, a failure is returned to the user.

[0144] In step 106, the full data of SHRAD and the distributed database are compared again, and the data marked as marked for deletion in the migration bucket is deleted from the distributed database.

[0145] In some embodiments, step 106 in the CEPH metadata migration and upgrade method based on a distributed database of the present application specifically includes:

[0146] Get the metadata of the object on SHARD and check whether the corresponding metadata exists in the distributed database;

[0147] If the corresponding metadata does not exist in the distributed database, the metadata of the object on SHARD is directly copied to the distributed database;

[0148] If the corresponding metadata exists in the distributed database, all SHARDs in the migration bucket are traversed;

[0149] Get metadata from the distributed database and check whether the corresponding metadata exists on SHARD;

[0150] If the corresponding metadata exists in SHARD, the data marked as deleted in the migration bucket on SHARD is deleted from the distributed database;

[0151] If the corresponding metadata does not exist in SHARD, the corresponding metadata whose mtime is less than the third time T3 is deleted from the distributed database.

[0152] For details, see Fig. 9 As shown, in this embodiment, the full amount is compared again, and the process of cleaning the marked deleted objects is as follows: the last round needs to be compared once, to ensure the data consistency of the final SHARD and the distributed database, and to prevent the problem of data inconsistency in the previous incremental comparison. In the last round of full comparison, the comparison is first performed from SHARD to the distributed database. Since the marked deletion is enabled while switching the single-read single-write database in the previous stage, all the data existing on SHARD must exist in the distributed database. Therefore, if there is some data on SHARD but not in the distributed database, it means that the data is omitted or mistakenly deleted in the migration, and it needs to be re-migrated to the distributed database. When comparing from the distributed database to SHARD, if it exists in the distributed database but not in SHARD, it is necessary to further determine whether the mtime of the metadata is less than T3. If it is, it is directly deleted. Otherwise, it means that the data is the metadata uploaded after switching the single-read single-write database. This further queries whether there is a marked deletion mark in the metadata. If it exists, it is deleted, otherwise the metadata is not deleted. Through the above process, the redundant data in the distributed database is deleted, and the data marked with the marked deletion is cleaned up to ensure that the data on the distributed database and SHARD are consistent before and after the migration.

[0153] In step 107, metadata in the INDEX pool of the SHARD is cleared.

[0154] Specifically, after all cluster data is migrated, the relevant storage pool is directly destroyed to release resources.

[0155] The following is a specific example of a CEPH metadata migration and upgrade method based on a distributed database of the present application:

[0156] See also Fig.10As shown, (1) the write mark of the migration bucket is modified to write to the SHARD and distributed database at the same time, and the uploaded, deleted, and modified metadata are updated to the SHRAD and distributed database at the same time; (2) the migration tool is used to migrate the existing metadata; (3) a comparison tool is used to perform a full comparison, and the time T1 of the full comparison is recorded, that is, the first moment; (4) the current time T2 is recorded, that is, the second moment, and the incremental data from T1 to T2 is compared, and the HEADER data of the migration bucket on the SHARD is migrated at the same time; (5) the current time T3 is recorded, that is, the third moment, and the incremental data from T2 to T3 is compared, and the HEADER data is migrated to the database. At the same time, the migration read-write state is switched to read database and write database, and the marked deletion is enabled; (6) the migration bucket data is fully compared, and the marked deleted data in the data is cleaned up; (7) the metadata in the INDEX pool in the original SHARD is cleaned up.

[0157] The CEPH metadata migration and upgrade method based on a distributed database of the present application is applicable to the scene where there is an upper limit on the number of objects in the object storage bucket, or when the user performs operations such as enumeration, query, deletion, and upload, the metadata-related latency consumption is large; this method can migrate the metadata in the user bucket from SHARD to the distributed database without interrupting the user's business, thereby achieving no upper limit on the number of objects stored in the object storage bucket and reducing the time consumption of related metadata operations.

[0158] In summary, the CEPH metadata migration and upgrade method based on distributed database has the following advantages: 1) The metadata in SHARD is migrated to the distributed database for storage, which avoids the problem that a single SHARD in the original SHARD cannot store too many records, and solves the problem of the limited storage quantity of a single bucket; through the orderliness of distributed database data storage and its own consistency characteristics, the PREPARE and COMPELETE operations done by the original process to ensure data consistency are reduced, the performance of user requests is improved, and the request latency is reduced. 2) The metadata stored in SHARD is migrated through the background migration threads on multiple gateways, making full use of the advantages of distributed storage, reducing migration time and improving migration efficiency. 3) In the dual-write phase, the ERROR log is used to record the metadata that failed to write or modify the distributed database, ensuring that the distributed database is available before switching to a single-read single-write database, avoiding the instability of the database service after the switch that affects user business. 4) Without interrupting user business, multiple rounds of comparison operations are performed to ensure the consistency of user data before and after migration, which not only protects user business, but also ensures the consistency of data before and after migration.

[0159] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0160] Reference Fig.11 , showing a structural block diagram of a CEPH metadata migration and upgrade device based on a distributed database of the present application, the device 1100 includes:

[0161] The first processing module 1101 is configured to, when the distributed database is available, modify the write flag of the migration bucket to double write, and update the latest uploaded / deleted / modified metadata to the SHRAD and the distributed database at the same time;

[0162] The second processing module 1102 is configured to migrate the stock metadata of SHRAD to the distributed database;

[0163] The third processing module 1103 is configured to perform a full data comparison between the SHRAD and the distributed database, and record the first time T1;

[0164] The fourth processing module 1104 is configured to compare the incremental data in the SHRAD and the distributed database during the period from the first time T1 to the second time T2, and simultaneously migrate the HEADER data of the migration bucket on the SHRAD to the distributed database;

[0165] The fifth processing module 1105 is configured to compare the incremental data in the SHRAD and the distributed database after the second time T2, and simultaneously migrate the HEADER data of the migration bucket on the SHRAD to the distributed database, switch the read-write state of the migration bucket to a single-read single-write database, and turn on the mark-delete flag of the migration bucket;

[0166] The sixth processing module 1106 is configured to perform full data comparison between the SHRAD and the distributed database again, and delete the data marked as marked for deletion in the migration bucket from the distributed database;

[0167] The seventh processing module 1107 is configured to clean up the metadata in the INDEX pool of the SHARD.

[0168] In some embodiments, the first processing module 1101 in the CEPH metadata migration and upgrade device based on a distributed database of the present application is specifically configured as follows:

[0169] First write the latest uploaded / deleted / modified metadata to SHRAD;

[0170] If writing to SHRAD fails, it returns failure directly without writing to the distributed database.

[0171] If the write to SHRAD is successful, the latest uploaded / deleted / modified metadata will continue to be written to the distributed database;

[0172] If writing to the distributed database fails, a new structure is added to record the object that failed to be written to the database, and success is returned.

[0173] In some embodiments, the second processing module 1102 in the CEPH metadata migration and upgrade device based on a distributed database of the present application is specifically configured to: traverse all SHARDs in the migration bucket, and directly migrate the original metadata of the current object in each SHARD to the distributed database.

[0174] In some embodiments, the third processing module 1103 in the CEPH metadata migration and upgrade device based on a distributed database of the present application is specifically configured as follows:

[0175] Record the current time node as the first moment T1;

[0176] Traverse the metadata of all objects on the SHARD in the migration bucket and check whether the corresponding metadata exists in the distributed database;

[0177] If the corresponding metadata does not exist in the distributed database, the metadata of the object on SHARD is directly copied to the distributed database;

[0178] If the corresponding metadata exists in the distributed database, compare the mtime in the metadata of the object on the SHARD with the mtime in the corresponding metadata on the distributed database to see if they are consistent;

[0179] If the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten to the distributed database.

[0180] In some embodiments, the fourth processing module 1104 in the CEPH metadata migration and upgrade device based on a distributed database of the present application is specifically configured as follows:

[0181] Record the current time node as the second moment T2;

[0182] Obtain the metadata of the object on SHARD, and determine whether mtine in the metadata of the object on SHARD is greater than the first moment T1;

[0183] If mtine is greater than the first moment T1, check whether the corresponding metadata exists in the distributed database;

[0184] If the corresponding metadata does not exist in the distributed database, the metadata of the object on the SHARD is directly copied to the distributed database; if the corresponding metadata exists in the distributed database, the mtime in the metadata of the object on the SHARD is compared with the mtime in the corresponding metadata on the distributed database to see if they are consistent; if the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten in the distributed database;

[0185] Traverse all SHARDs in the migration bucket;

[0186] Obtain all metadata whose mtine is greater than the first moment T1 from the distributed database, and check whether the corresponding metadata exists in SHARD;

[0187] If the corresponding metadata does not exist in SHARD, the corresponding metadata in the distributed database will be deleted; if the corresponding metadata exists in SHARD, the metadata of the object on SHARD will be migrated to the distributed database, and finally the HEADER data of the migration bucket on SHARD will be migrated to the distributed database.

[0188] In some embodiments, the fifth processing module 1105 in the CEPH metadata migration and upgrade device based on a distributed database of the present application is specifically configured as follows:

[0189] Compare the incremental data in SHRAD and the distributed database after the second time T2;

[0190] After the incremental comparison is completed, lock the migration bucket;

[0191] Switch the read / write state of the migration bucket to a single-read single-write database, and record the current switching time as the third moment T3;

[0192] Migrate the HEADER data of the migration bucket on SHARD to the distributed database;

[0193] Enable the delete flag of the migration bucket;

[0194] Unlock the migration bucket.

[0195] In some embodiments, the sixth processing module 1106 in the CEPH metadata migration and upgrade device based on a distributed database of the present application is specifically configured as follows:

[0196] Get the metadata of the object on SHARD and check whether the corresponding metadata exists in the distributed database;

[0197] If the corresponding metadata does not exist in the distributed database, the metadata of the object on SHARD is directly copied to the distributed database;

[0198] If the corresponding metadata exists in the distributed database, all SHARDs in the migration bucket are traversed;

[0199] Get metadata from the distributed database and check whether the corresponding metadata exists on SHARD;

[0200] If the corresponding metadata exists in SHARD, the data marked as deleted in the migration bucket on SHARD is deleted from the distributed database;

[0201] If the corresponding metadata does not exist in SHARD, the corresponding metadata whose mtime is less than the third time T3 is deleted from the distributed database.

[0202] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0203] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0204] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0205] Fig.12 1 is a block diagram of an electronic device 1200 shown in the present application. For example, the electronic device 1200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0206] Reference Fig.12, the electronic device 1200 may include one or more of the following components: a processing component 1202 , a memory 1204 , a power component 1206 , a multimedia component 1208 , an audio component 1210 , an input / output (I / O) interface 1212 , a sensor component 1214 , and a communication component 1216 .

[0207] The processing component 1202 generally controls the overall operation of the electronic device 1200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 1202 may include one or more modules to facilitate the interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate the interaction between the multimedia component 1208 and the processing component 1202.

[0208] The memory 1204 is configured to store various types of data to support operations on the device 1200. Examples of such data include instructions for any application or method operating on the electronic device 1200, contact data, phone book data, messages, images, videos, etc. The memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0209] The power supply component 1206 provides power to the various components of the electronic device 1200. The power supply component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1200.

[0210] The multimedia component 1208 includes a screen that provides an output interface between the electronic device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the device 1200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0211] The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC), and when the electronic device 1200 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1204 or sent via the communication component 1216. In some embodiments, the audio component 1210 also includes a speaker for outputting audio signals.

[0212] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0213] The sensor assembly 1214 includes one or more sensors for providing various aspects of status assessment for the electronic device 1200. For example, the sensor assembly 1214 can detect the open / closed state of the device 1200, the relative positioning of components, such as the display and keypad of the electronic device 1200, and the sensor assembly 1214 can also detect the position change of the electronic device 1200 or a component of the electronic device 1200, the presence or absence of user contact with the electronic device 1200, the orientation or acceleration / deceleration of the electronic device 1200, and the temperature change of the electronic device 1200. The sensor assembly 1214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0214] The communication component 1216 is configured to facilitate wired or wireless communication between the electronic device 1200 and other devices. The electronic device 1200 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 8G), or a combination thereof. In an exemplary embodiment, the communication component 1216 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0215] In an exemplary embodiment, the electronic device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0216] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, and the instructions can be executed by a processor 1220 of the electronic device 1200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0217] Fig.13 19 is a block diagram of a computer storage medium 1900 shown in the present application. For example, the computer storage medium 1900 may be provided as a server.

[0218] Reference Fig.13 , the computer storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0219] The computer storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0220] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0221] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0222] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0223] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0224] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0225] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0226] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0228] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0229] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A CEPH metadata migration and upgrade method based on a distributed database, characterized in that: The method comprises: If the distributed database is available, change the write flag of the migration bucket to double write, and update the latest uploaded / deleted / modified metadata to SHRAD and the distributed database at the same time; Migrate SHRAD's existing metadata to a distributed database; Compare the full data of SHRAD and distributed databases, and record the first moment T1; Compare the incremental data in SHRAD and the distributed database from the first time T1 to the second time T2, and migrate the HEADER data of the migration bucket on SHRAD to the distributed database at the same time; Compare the incremental data in SHRAD and the distributed database after the second time T2, and migrate the HEADER data of the migration bucket on SHRAD to the distributed database at the same time, switch the read-write state of the migration bucket to a single-read single-write database, and enable the mark-delete flag of the migration bucket; Compare the full data between SHRAD and the distributed database again, and delete the data marked for deletion in the migration bucket from the distributed database. Clean up the metadata in the SHARD INDEX pool.

2. According to a distributed database-based CEPH metadata migration and upgrade method according to claim 1, it is characterized in that: Update the latest uploaded / deleted / modified metadata to SHRAD and distributed databases simultaneously, including: First write the latest uploaded / deleted / modified metadata to SHRAD; If writing to SHRAD fails, it returns failure directly without writing to the distributed database. If the write to SHRAD is successful, the latest uploaded / deleted / modified metadata will continue to be written to the distributed database; If writing to the distributed database fails, a new structure is added to record the object that failed to be written to the database, and success is returned.

3. According to a distributed database-based CEPH metadata migration and upgrade method according to claim 2, it is characterized in that: Migrate the existing metadata of SHRAD to the distributed database. Specifically, traverse all SHARDs in the migration bucket and directly migrate the original metadata of the current object in each SHARD to the distributed database.

4. According to a distributed database-based CEPH metadata migration and upgrade method according to claim 3, it is characterized in that: A full data comparison between SHRAD and distributed databases is performed, including: Record the current time node as the first moment T1; Traverse the metadata of all objects on the SHARD in the migration bucket and check whether the corresponding metadata exists in the distributed database; If the corresponding metadata does not exist in the distributed database, the metadata of the object on SHARD is directly copied to the distributed database; If the corresponding metadata exists in the distributed database, compare the mtime in the metadata of the object on the SHARD with the mtime in the corresponding metadata on the distributed database to see if they are consistent; If the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten to the distributed database.

5. A CEPH metadata migration and upgrade method based on a distributed database according to claim 4, characterized in that: Compare the incremental data in SHRAD and the distributed database from the first time T1 to the second time T2, and migrate the HEADER data of the migration bucket on SHRAD to the distributed database at the same time, including: Record the current time node as the second moment T2; Obtain the metadata of the object on SHARD, and determine whether mtine in the metadata of the object on SHARD is greater than the first moment T1; If mtine is greater than the first moment T1, check whether the corresponding metadata exists in the distributed database; If the corresponding metadata does not exist in the distributed database, the metadata of the object on the SHARD is directly copied to the distributed database; if the corresponding metadata exists in the distributed database, the mtime in the metadata of the object on the SHARD is compared with the mtime in the corresponding metadata on the distributed database to see if they are consistent; if the mtime of the two is inconsistent, the metadata of the object on the SHARD is directly overwritten in the distributed database; Traverse all SHARDs in the migration bucket; Obtain all metadata whose mtine is greater than the first moment T1 from the distributed database, and check whether the corresponding metadata exists in SHARD; If the corresponding metadata does not exist in SHARD, the corresponding metadata in the distributed database will be deleted; if the corresponding metadata exists in SHARD, the metadata of the object on SHARD will be migrated to the distributed database, and finally the HEADER data of the migration bucket on SHARD will be migrated to the distributed database.

6. A CEPH metadata migration and upgrade method based on a distributed database according to claim 5, characterized in that: Compare the incremental data in SHRAD and the distributed database after the second time T2, and migrate the HEADER data of the migration bucket on SHRAD to the distributed database at the same time, switch the read-write state of the migration bucket to a single-read single-write database, and enable the mark-delete flag of the migration bucket, including: Compare the incremental data in SHRAD and the distributed database after the second time T2; After the incremental comparison is completed, lock the migration bucket; Switch the read / write state of the migration bucket to a single-read single-write database, and record the current switching time as the third moment T3; Migrate the HEADER data of the migration bucket on SHARD to the distributed database; Enable the delete flag of the migration bucket; Unlock the migration bucket.

7. A CEPH metadata migration and upgrade method based on a distributed database according to claim 6, characterized in that: Compare the full data between SHRAD and the distributed database again, and delete the data marked for deletion in the migration bucket from the distributed database. Specifically: Get the metadata of the object on SHARD and check whether the corresponding metadata exists in the distributed database; If the corresponding metadata does not exist in the distributed database, the metadata of the object on the SHARD is directly copied to the distributed database; if the corresponding metadata exists in the distributed database, all SHARDs in the migration bucket are traversed; metadata is obtained from the distributed database, and then it is checked whether the corresponding metadata exists on the SHARD; if the corresponding metadata exists in the SHARD, the data marked as deleted in the migration bucket on the SHARD is deleted from the distributed database; if the corresponding metadata does not exist in the SHARD, the corresponding metadata whose mtime is less than the third moment T3 is deleted from the distributed database.

8. A CEPH metadata migration and upgrade device based on a distributed database, characterized in that: The device comprises: The first processing module is configured to, when the distributed database is available, modify the write flag of the migration bucket to double write, and update the latest uploaded / deleted / modified metadata to SHRAD and the distributed database at the same time; A second processing module is configured to migrate the stock metadata of SHRAD to a distributed database; The third processing module is configured to compare the full amount of data between SHRAD and the distributed database, and record the first time T1; The fourth processing module is configured to compare the incremental data in the SHRAD and the distributed database during the period from the first time T1 to the second time T2, and simultaneously migrate the HEADER data of the migration bucket on the SHRAD to the distributed database; The fifth processing module is configured to compare the incremental data in the SHRAD and the distributed database after the second time T2, and simultaneously migrate the HEADER data of the migration bucket on the SHRAD to the distributed database, switch the read-write state of the migration bucket to a single-read single-write database, and turn on the mark-delete flag of the migration bucket; The sixth processing module is configured to perform full data comparison between the SHRAD and the distributed database again, and delete the data marked as marked for deletion in the migration bucket from the distributed database; The seventh processing module is configured to clean up metadata in the INDEX pool of SHARD.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the CEPH metadata migration and upgrade method based on a distributed database as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the CEPH metadata migration and upgrade method based on a distributed database as described in any one of claims 1 to 7 is implemented.

Images