Data synchronization method, device, computer equipment and storage medium

Abstract

The present application relates to a data synchronization method, apparatus, computer equipment and storage medium, which obtains the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration phase; synchronizes the database metadata according to the forward comparison result and reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata. The above scheme compares the first metadata in the object index pool and the second metadata in the database in stages, and synchronizes the first metadata to the database, that is, uses the database to store metadata, does not involve bucket storage, and thus solves the problem caused by too many or too few shards in the bucket; and ensures that the first metadata in the object index pool is stored in the database, and there is no dirty data.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data synchronization method, apparatus, computer equipment and storage medium. Background Art

[0002] Metadata can be understood as data that describes data attributes, and has functions such as recording storage locations, recording historical data, and data search. With the rapid development of computer technology, various types of data have shown explosive growth, and the management and storage of metadata for various types of data are also facing new challenges.

[0003] In traditional technology, a new bucket is usually created to store object data, and the bucket is divided into a specified number of shards to store metadata related to the object data. For example, when object data is uploaded, a consistent hashing algorithm is used to calculate the target shard of the metadata related to the object data, and the target shard is used to store the metadata.

[0004] However, if there are too few shards in the bucket, the concurrent data storage capacity will be insufficient; if there are too many shards, the object data listing delay will be long; and if the number of shards is adjusted dynamically, business interruption will occur. Summary of the invention

[0005] Based on this, it is necessary to provide a data synchronization method, device, computer equipment and storage medium to address the above technical problems, which can solve the problems caused by too many or too few shards in the bucket and dynamic adjustment of the number of shards.

[0006] In a first aspect, the present application provides a data synchronization method, the method comprising:

[0007] Obtaining first metadata corresponding to each shard of a target bucket in an object index pool in a target migration phase; wherein the target migration phase is a first migration phase, a second migration phase, or a third migration phase, the end time of the first migration phase is the start time of the second migration phase, and the end time of the second migration phase is the start time of the third migration phase;

[0008] The database is synchronized with metadata according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata.

[0009] In one of the embodiments, when the target migration phase is the first migration phase, the first metadata is metadata stored in the object index pool before the start time of the first migration phase;

[0010] The step of synchronizing the database with metadata according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database includes:

[0011] Comparing the first metadata with metadata stored in the database before the start time of the first migration phase to obtain a positive comparison result;

[0012] When the forward comparison result is that the first target metadata in the first metadata does not exist in the database, synchronizing the first target metadata to the database;

[0013] Comparing all metadata stored in the database with the first metadata to obtain a reverse comparison result;

[0014] When the reverse comparison result is that the second target metadata exists in the database and the second target metadata does not exist in the first metadata, the second target metadata is deleted from the database.

[0015] In one of the embodiments, when the target migration phase is the second migration phase, the first metadata is metadata stored in the object index pool between the start time of the first migration phase and the end time of the first migration phase;

[0016] The step of synchronizing the database with metadata according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database includes:

[0017] Comparing the first metadata with metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a forward comparison result;

[0018] When the forward comparison result is that the third target metadata in the first metadata does not exist in the database, synchronizing the third target metadata to the database;

[0019] Comparing the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase with the first metadata to obtain a reverse comparison result;

[0020] When the reverse comparison result shows that the fourth target metadata exists in the database and the fourth target metadata does not exist in the first metadata, deleting the fourth target metadata from the database;

[0021] After the metadata for non-read operations in the target bucket is locked, the header data of each shard of the target bucket is migrated to the database.

[0022] In one of the embodiments, in the process of synchronizing the metadata of the database according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database, the method further includes:

[0023] In the second migration phase, a deletion mark is added to the metadata to be deleted;

[0024] The metadata to be deleted is metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is metadata common to the database and the object index pool.

[0025] In one of the embodiments, when the target migration stage is the third migration stage, the first metadata is all metadata stored in the object index pool up to the end time of the second migration stage;

[0026] The step of synchronizing the database with metadata according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database includes:

[0027] Comparing the first metadata with all metadata stored in the database to obtain a positive comparison result;

[0028] When the forward comparison result is that the fifth target metadata in the first metadata does not exist in the database, synchronizing the fifth target metadata to the database; wherein the update time of the fifth target metadata is less than the start time of the third migration stage;

[0029] Comparing all metadata stored in the database with the first metadata to obtain a reverse comparison result;

[0030] If the reverse comparison result shows that the sixth target metadata exists in the database and the sixth target metadata does not exist in the first metadata, the sixth target metadata is deleted; wherein the update time of the sixth target metadata is less than the start time of the third migration stage;

[0031] The metadata to be deleted that is marked for deletion in the database is deleted.

[0032] In one embodiment, the method further comprises:

[0033] When the forward comparison result is that the first metadata exists in the second metadata, the metadata with the latest update time between the first metadata and the second metadata is synchronized to the database.

[0034] In one embodiment, the method further comprises:

[0035] Obtaining the number of threads used for comparing the first metadata with the second metadata;

[0036] Determine, according to the number of shards of the target bucket in the object index pool and the number of threads, a shard allocated to each thread for comparison with the second metadata;

[0037] Control each thread to perform a forward comparison operation on the first metadata in the corresponding shard and the second metadata to obtain a forward comparison result;

[0038] Determining, according to the number of the second metadata and the number of threads, second metadata allocated to each thread for comparison with the first metadata;

[0039] Each thread is controlled to perform a reverse comparison operation on the corresponding second metadata and the first metadata to obtain a reverse comparison result.

[0040] In a second aspect, the present application further provides a data synchronization device, the device comprising:

[0041] An acquisition module, used to acquire first metadata corresponding to each shard of a target bucket in an object index pool in a target migration phase; wherein the target migration phase is a first migration phase, a second migration phase, or a third migration phase, the end time of the first migration phase is the start time of the second migration phase, and the end time of the second migration phase is the start time of the third migration phase;

[0042] A synchronization module, used to synchronize metadata on the database according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata.

[0043] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0044] Obtaining first metadata corresponding to each shard of a target bucket in an object index pool in a target migration phase; wherein the target migration phase is a first migration phase, a second migration phase, or a third migration phase, the end time of the first migration phase is the start time of the second migration phase, and the end time of the second migration phase is the start time of the third migration phase;

[0045] The database is synchronized with metadata according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0047] Obtaining first metadata corresponding to each shard of a target bucket in an object index pool in a target migration phase; wherein the target migration phase is a first migration phase, a second migration phase, or a third migration phase, the end time of the first migration phase is the start time of the second migration phase, and the end time of the second migration phase is the start time of the third migration phase;

[0048] The database is synchronized with metadata according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata.

[0049] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, which implements the following steps when executed by a processor:

[0050] Obtaining first metadata corresponding to each shard of a target bucket in an object index pool in a target migration phase; wherein the target migration phase is a first migration phase, a second migration phase, or a third migration phase, the end time of the first migration phase is the start time of the second migration phase, and the end time of the second migration phase is the start time of the third migration phase;

[0051] The database is synchronized with metadata according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata.

[0052] The above-mentioned data synchronization method, device, computer equipment and storage medium obtain the first metadata corresponding to each shard of the target bucket in the object index pool in the target migration stage; wherein the target migration stage is the first migration stage, the second migration stage or the third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage; according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database, the database is synchronized with the metadata; wherein the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata. The above-mentioned scheme compares the first metadata in the object index pool and the second metadata in the database in stages, and synchronizes the first metadata to the database, that is, the database is used to store metadata, and bucket storage is not involved, thereby solving the problem caused by too many or too few shards in the bucket; and through forward comparison and reverse comparison, it can be ensured that the first metadata in the object index pool is stored in the database, and there will be no dirty data in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of a data synchronization method in an embodiment;

[0054] Figure 2 A schematic diagram of storing metadata in an object index pool in one embodiment;

[0055] Figure 3 A schematic diagram of a process of synchronizing metadata on a database in one embodiment;

[0056] Figure 4 A schematic diagram of a forward alignment in the first migration stage in one embodiment;

[0057] Figure 5 A schematic diagram of reverse alignment in the first migration stage in one embodiment;

[0058] Figure 6 It is a schematic diagram of another process of synchronizing metadata on a database in one embodiment;

[0059] Figure 7 is a schematic diagram of a forward alignment in the second migration stage in one embodiment;

[0060] Figure 8 is a schematic diagram of reverse alignment in the second migration stage in one embodiment;

[0061] Fig. 9 A schematic diagram of locking and migrating header data in one embodiment;

[0062] Fig.10 A schematic diagram of a process for synchronizing metadata on a database in another embodiment;

[0063] Fig.11 is a schematic diagram of a forward alignment in the third migration stage in one embodiment;

[0064] Fig.12 A schematic diagram of reverse alignment in the third migration stage in one embodiment;

[0065] Fig.13 A schematic diagram of cleaning up metadata to be deleted that is marked for deletion in one embodiment;

[0066] Fig.14 A schematic diagram of a flow chart of using multiple threads to execute a data comparison task in one embodiment;

[0067] Fig.15 A schematic diagram of allocating slices to each thread in one embodiment;

[0068] Fig.16 A schematic diagram of allocating second metadata to each thread in one embodiment;

[0069] Fig.17 A schematic diagram of another data synchronization process in one embodiment;

[0070] Fig.18 is a structural block diagram of a data synchronization device in an embodiment;

[0071] Fig.19 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0073] The data synchronization method provided in the embodiment of the present application can be applied to the application scenario of migrating metadata stored in the object index pool to a distributed database. The method can be executed by a server or by a terminal with a certain computing power.

[0074] The server may be implemented as an independent server or a server cluster composed of multiple servers. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0075] With the rapid development of the Internet, Internet of Things, cloud computing and big data, data is growing explosively. The data generated by various applications such as social networks, mobile communications, online video and audio, e-commerce, sensor networks and scientific experiments not only have large storage capacity, but also have significant characteristics such as a variety of data types, large changes in data size and fast flow. They can often generate massive files in the hundreds of millions or even billions or tens of billions, which faces huge challenges in metadata management. In the current storage process of object storage services, there are the following three problems with object metadata management.

[0076] 1. When creating a new bucket for storing data, the bucket is divided into a specified number of shards to store object-related metadata information, which is stored in an ObjectMap Key-Value (omap(kv)) structure. A shard corresponds to a Reliable Autonomic Distributed ObjectStore (rados) object. A rados object consists of an object flag, object data, and object metadata. The object flag is used to uniquely identify an object; the object data is used to correspond to a file in the local file system, and the object data is stored in this file; the object metadata is stored in the form of a key-value pair.

[0077] When uploading a file object, the consistent hashing algorithm can be used to calculate the target shard of the object metadata and store the metadata information of the file object. The consistent hashing algorithm ensures that each shard can carry the file object metadata as evenly as possible. However, as the number of objects in the bucket increases, the metadata entries stored on each shard will increase accordingly. When the alarm threshold specified by the cluster is reached, a large number of object metadata mapping (large omap) alarms will appear. At this time, a large number of concurrent requests for the bucket will increase the number of requests falling on the same shard. The consistency of Rados object data is guaranteed by a single thread, so it will directly affect the execution efficiency of user operations such as uploading object data, downloading object data, and deleting object data.

[0078] 2. The cluster supports the reshard operation for the original bucket, that is, increasing the number of shards of the bucket and rebalancing the data at the same time, so that the number of metadata (omap) entries stored on a single shard is reduced. When concurrent requests are accessed, since the data is balanced on more shards, the blocking phenomenon of requests to a single shard will be further reduced, thereby improving concurrent performance. However, in the process of adjusting the shards of the bucket, some business interruptions may occur due to the migration of data involved.

[0079] 3. If the number of shards is increased when a new bucket is created, new problems will also be introduced. When the number of files in the bucket reaches the billion level, the number of omap entries stored on each shard will increase. For a single shard, the object metadata key value (omap_key) managed on it is ordered, but the data between shards is not ordered. Therefore, when a request to list 1,000 objects in batches is issued, the processing logic is to obtain data of a certain entry from each shard, and then merge and sort all the listed data, and select the top 1,000 entries in lexicographic order as the response result. Therefore, when there are too many shards, the listing performance will be significantly reduced.

[0080] Based on this, the embodiment of this application introduces the Advanced Distributed SQL Database (TiDB) to manage object metadata, that is, to migrate the metadata in the object index pool to the TiDB database. Among them, TiDB is an open source distributed relational database, and a hybrid distributed database that supports both online transaction processing and online analytical processing. It has the following advantages:

[0081] 1. TiDB database is a distributed storage database that can achieve replica-level disaster recovery backup, with 3 replicas by default.

[0082] 2. TiDB database supports massive data and high-concurrency online transaction processing (OLTP) scenarios, and there is no need to worry about the problem of concurrent performance degradation caused by shard resource preemption and competition.

[0083] 3. The TiDB database has an architecture design that separates storage and computing. It can expand or shrink computing and storage online as needed, and the expansion or reduction process is transparent to application operation and maintenance personnel.

[0084] 4. The key values ​​of the TiDB database are strictly ordered globally. Based on this feature, the concept of sharding can be completely abandoned, and the performance of data enumeration will be significantly improved.

[0085] The following is a detailed introduction to the various implementation steps of the data synchronization method provided in the embodiment of the present application.

[0086] In one embodiment, Figure 1 As shown, a data synchronization method is provided, which is described by taking the method applied to a server as an example, and includes the following steps:

[0087] S101, obtaining first metadata corresponding to each shard of a target bucket in an object index pool during a target migration phase.

[0088] For example, see Figure 2 , Figure 2 A schematic diagram of storing metadata in an object index pool is provided. The object index pool includes multiple buckets, each of which can be divided into multiple shards, and the target bucket is any bucket in the object index pool. Metadata is stored in the omap(kv) structure. Taking shard 1 and shard 2 as examples, omap_key is the metadata key value, and type is the type of omap_key. Shard 1 stores three single-version object metadata, namely single_obj1, single_obj2, and single_obj3; the single-version object metadata types are all plain types; and shard 2 stores two versions of a file, among which ver_obj is of plain type, which has no practical use and exists as a placeholder, and ver_obj_1 and ver_obj_2 are also of plain type, which represent the placeholders of these two versions respectively. When all version information of this object is listed, ver_obj_1 and ver_obj_2 will be listed. The types of ver_obj_1_1000 and ver_obj_2_1000 are instance types, which represent the instance metadata identification information corresponding to the specific version, that is, the specific instance corresponding to ver_obj_1 is ver_obj_1_1000. The last ver_obj_1_1001 type is the latest version (olh) type; it points to the latest version information of the object. When the version number is not specified, the latest version information of the object will be found according to the olh information. The statistical information of each shard is stored in the header data (header) attribute of the shard. For example, the statistical information of shard 1 is stored in the shard1_header attribute of the shard's header data; the statistical information of shard 2 is stored in the shard2_header attribute of the shard's header data. Among them, the header records the statistical information such as how much object data and capacity are stored on the shard. How many objects are in a bucket and how much capacity they occupy are statistically summarized by the collected shard header information.

[0089] The target migration stage is the first migration stage, the second migration stage or the third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage. For example, assuming that time T0 is the start time of the first migration stage and T1 is the end time of the first migration stage, that is, the time period between time T0 and time T1 is the first migration stage; the start time of the second migration stage is time T1, assuming that the end time of the second migration stage is time T2, that is, the time period between time T1 and time T2 is the second migration stage; the start time of the third migration stage is time T2, and the time period after time T2 is the third migration stage.

[0090] Optionally, you can introduce a data structure comparison object compare_obj, which records the start comparison time last_compare_time and the end comparison time current_compare_time as the two ends of the migration phase, in microseconds (μs). At the same time, a new whether to suspend flag is_suspend(bool) and a suspension duration flag interrupt_sec(s) are added to record whether to suspend and the suspension duration (seconds), respectively. When the comparison task puts pressure on the cluster and causes an increase in the latency of normal business, the comparison task can be suspended at any time, and the comparison business can be resumed when the cluster business pressure is not large.

[0091] For example, compare_obj is initialized as follows:

[0092] {

[0093] "last_compare_time":"0.000000",

[0094] "current_compare_time":"0.000000",

[0095] "is_suspend": "false",

[0096] "interrupt_sec": 0

[0097] }

[0098] Exemplarily, the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration phase is stored in the disk, and the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration phase can be read into the memory to perform data comparison operations.

[0099] S102: Synchronize metadata on the database according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database.

[0100] The forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0101] Exemplarily, metadata synchronization can be performed on the database first according to the forward comparison result, for example, the first metadata not included in the second metadata of the database can be synchronized to the database; and then metadata synchronization can be performed on the database according to the reverse comparison result, for example, the second metadata not included in the first metadata of the object index pool is regarded as dirty data and the dirty data is deleted from the database. The database can be a TiDB database.

[0102] The above data synchronization method obtains the first metadata corresponding to each shard of the target bucket in the object index pool in the target migration stage; wherein the target migration stage is the first migration stage, the second migration stage or the third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage; according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database, the database is synchronized with the metadata; wherein the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata. The above scheme compares the first metadata in the object index pool and the second metadata in the database in stages, and synchronizes the first metadata to the database, that is, the database is used to store metadata, and bucket storage is not involved, thereby solving the problem caused by too many or too few shards in the bucket; and, through forward comparison and reverse comparison, it can be ensured that the first metadata in the object index pool is stored in the database, and there will be no dirty data in the database.

[0103] In some optional implementations, the embodiments of the present application are described by taking the target migration phase as the first migration phase as an example. Assuming that the first migration phase is the time period between time T0 and time T1, the first metadata to be compared is the metadata stored in the object index pool before the start time of the first migration phase, that is, the metadata before the start time T0.

[0104] Based on this, see Figure 3 , Figure 3 A schematic diagram of a process for synchronizing metadata of a database is provided, which specifically includes the following steps:

[0105] S301, comparing the first metadata with the metadata stored in the database before the start time of the first migration phase to obtain a forward comparison result.

[0106] Exemplarily, the first migration phase starts at compare_obj as follows:

[0107] {

[0108] "last_compare_time": "T0",

[0109] "current_compare_time":"0.000000",

[0110] "is_suspend": "false",

[0111] "interrupt_sec": 0

[0112] }

[0113] At this time, the second metadata is the metadata stored in the database before the start time of the first migration phase. The metadata stored in the database before the start time of the first migration phase can be read into the memory, and the first metadata is compared with the metadata stored in the database before the start time of the first migration phase to obtain a positive comparison result.

[0114] S302: When the forward comparison result is that the first target metadata in the first metadata does not exist in the database, synchronize the first target metadata to the database.

[0115] Furthermore, if the result of the forward comparison is that the first target metadata in the first metadata does not exist in the database, the full amount of data of the first target metadata is copied to the database. If the result of the forward comparison is that the first target metadata in the first metadata exists in the database, the update time mtime (recorded in the metadata attribute field) of the metadata in the object index pool and the database can be compared, and the metadata with the latest update time mtime is used as the standard, and then the metadata with the latest update time mtime is stored in the database.

[0116] For example, see Figure 4 , Figure 4A schematic diagram of the forward comparison in the first migration phase is provided. Among them, the data update time of the metadata key values ​​Obj1, Obj4 and Obj801 included in shard 1 is less than T0, and the data update time of the metadata key values ​​Obj3, Obj6, and Obj800 included in shard 2 is less than T0, and they can all be used as the first metadata; when the forward comparison result is that these data do not exist in the database, these data are synchronized to the database. Among them, Obj2 and Obj5 are deleted data and do not need to be synchronized to the database. The update time of Obj7 and Obj8 is greater than T0, and they are changed data. Obj7 and Obj8 do not need to be processed in the first migration phase. The update time of Obj804 and Obj802 is greater than T0, and they are written data. Since the first migration phase supports data being written to the object index pool and the database at the same time, the metadata corresponding to Obj804 and Obj802 will be written to the database; and the read data is only read in the object index pool.

[0117] S303: compare all metadata stored in the database with the first metadata to obtain a reverse comparison result.

[0118] Furthermore, all metadata stored in the database may be extracted into the memory, and all metadata stored in the database may be compared with the first metadata to obtain a reverse comparison result.

[0119] S304: When the reverse comparison result shows that the second target metadata exists in the database and the second target metadata does not exist in the first metadata, the second target metadata is deleted from the database.

[0120] Furthermore, when the reverse comparison result shows that the second target metadata exists in the database and does not exist in the first metadata, the second target metadata is considered to be dirty data and can be deleted from the database.

[0121] For example, see Figure 5 , Figure 5 A schematic diagram of the reverse comparison in the first migration phase is provided. Among them, if the update time of Obj34, Obj41 and Obj502 is less than T0 and exists in the first metadata, no processing will be performed. If the update time of Obj43 is greater than T0, Obj43 will not be processed in the first migration phase. If the update time of Obj107 and Obj504 is less than T0 and does not exist in the first metadata, they are considered dirty data, and Obj107 and Obj504 can be deleted from the database.

[0122] Furthermore, after all the second metadata in the database are traversed, last_compare_time, i.e. T0, is assigned to current_compare_time and persistence is completed. That is, the update time of the data compared in the first migration phase is before T0, and new data is generated during the comparison process, and the update time must be greater than T0. This part of data will be left for the second migration phase to process. Thus, the comparison process of the entire first migration phase is completed.

[0123] Then compare_obj can be updated as follows:

[0124] {

[0125] "last_compare_time": "T0",

[0126] "current_compare_time":"T0",

[0127] "is_suspend": "false",

[0128] "interrupt_sec": 0

[0129] }

[0130] In the embodiment of the present application, in the first migration phase, the first metadata stored in the object index pool before the start time of the first migration phase is migrated to the database, completing the migration of the stock data before the start time of the first migration phase.

[0131] In some optional implementations, the embodiment of the present application is described by taking the target migration stage as the second migration stage as an example. Assuming that the second migration stage is the time period between time T1 and time T2, the first metadata to be compared is the metadata stored in the object index pool between the start time of the first migration stage and the end time of the first migration stage, that is, the metadata between time T1 and time T2.

[0132] Based on this, see Figure 6 , Figure 6 Another flowchart for synchronizing metadata of a database is provided, which specifically includes the following steps:

[0133] S601 : Compare the first metadata with the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a forward comparison result.

[0134] Exemplarily, the second migration phase starts at compare_obj as follows:

[0135] {

[0136] "last_compare_time": "T0",

[0137] "current_compare_time":"T1",

[0138] "is_suspend": "false",

[0139] "interrupt_sec": 0

[0140] }

[0141] At the beginning of the second migration phase, data can still be written to the object index pool and the database at the same time, while data can only be read in the object index pool. In addition, the target bucket mark deletion switch is turned on, that is, a mark field mark_del is added to the attribute data of the target bucket. The default value is false, which is changed to true and persisted to the bottom layer; after that, the deletion operation of the internal objects of the target bucket will delete the real data, but the mark_del in the metadata attribute in the TiDB database will be marked as true, that is, the metadata in the TiDB database is only marked for deletion, not actually deleted, while the metadata in the object index pool will be actually deleted.

[0142] At this time, the second metadata is the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase. The metadata stored in the object index pool between the start time of the first migration phase and the end time of the first migration phase can be extracted into the memory, and the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase can be extracted into the memory, and the first metadata is compared with the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a positive comparison result.

[0143] S602: When the forward comparison result is that the third target metadata in the first metadata does not exist in the database, synchronize the third target metadata to the database.

[0144] Exemplarily, when the forward comparison result is that the third target metadata in the first metadata does not exist in the database, the third target metadata can be synchronized to the database. When the forward comparison result is that the third target metadata in the first metadata exists in the database, the update time mtime of the metadata in the object index pool and the database can be compared, and the metadata with the latest update time mtime is used as the standard, and then the metadata with the latest update time mtime is stored in the database.

[0145] For example, see Figure 7 , Figure 7A schematic diagram of the forward comparison in the second migration phase is provided. Among them, Obj5 is the stock data before T0, and the deletion is executed after T1, so the real deletion is completed in the object index pool, but the deletion is marked in the database and the mtime is updated to be greater than T1, and it is not actually deleted. Obj7 is the update of the stock data, so the update of the data in the database and the object index pool is completed during the comparison process, and the update time mtime is greater than T1, so Obj7 does not belong to the comparison metadata in this phase. Obj800 is the new data added by the business, and it is in dual-write mode at this time, so an object metadata record is added to the database and the object index pool. Therefore, only Obj4 is the metadata uploaded after the start of the first migration phase (after T0), and the write fails when writing to the database, so the metadata can be migrated from the object index pool to the database. Among them, Obj1, Obj3, Obj803 and Obj804 are all stock data that already exist in the database.

[0146] S603: Compare the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase with the first metadata to obtain a reverse comparison result.

[0147] Furthermore, the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase may be reversely compared with the first metadata to obtain a reverse comparison result.

[0148] S604: When the reverse comparison result shows that the fourth target metadata exists in the database and the fourth target metadata does not exist in the first metadata, the fourth target metadata is deleted from the database.

[0149] Furthermore, when the reverse comparison result shows that the fourth target metadata exists in the database and does not exist in the first metadata, the fourth target metadata is considered to be dirty data and can be deleted from the database.

[0150] See also Figure 8 , Figure 8 A schematic diagram of the reverse comparison of the second migration phase is provided. Among them, the deletion time of metadata Obj41 is after T1, the database marks Obj41 as deleted and updates the time to be greater than T1, while Obj41 in the object index pool is actually deleted. The newly added metadata Obj43 of the business is still added to the database and the object index pool at the same time; the metadata (Obj107, Obj504) that does not exist in the object index pool but exists in the database, that is, the fourth target metadata, is regarded as dirty data and cleared from the database; the updated metadata Obj741 is still updated simultaneously in the database and the object index pool. Obj34 and Obj502 are metadata that exist in both the database and the object index pool and can be left unprocessed.

[0151] S605, after locking the metadata for non-read operations in the object index pool, migrate the header data of each shard of the target bucket in the object index pool to the database.

[0152] For example, since the metadata is not locked, a phased comparison migration method is adopted during the metadata migration process in consideration of the final data consistency; however, the target bucket must be locked when migrating the shard header, so that before the header migration is completed, all add, delete and modify operations on the target bucket are not allowed, because the header statistics information read into the memory will synchronously modify the underlying header information during the add, delete and modify operations of the object, resulting in a mismatch between the header information in the memory and the disk. Direct migration to the database without locking will cause the header statistics in the database to be inconsistent with the actual situation.

[0153] Furthermore, the header data of each shard in the target bucket can be further migrated. Before migrating the header, the metadata for non-read operations in the target bucket needs to be locked, that is, adding, deleting, and modifying objects in the target bucket are not allowed during the header migration process. After locking the metadata for non-read operations in the object index pool, the header data of each shard of the target bucket in the object index pool is migrated to the database. Since the number of shards of a bucket is usually 2048, each shard corresponds to a header statistic, the number is very small, and the migration can be completed within 1 second, so in order to ensure the consistency of the header data, it is necessary to sacrifice short-term business.

[0154] See also Fig. 9 , Fig. 9 A schematic diagram of locking the migration header data is provided. In the process of migrating the header, it is still in dual-write and single-read mode, that is, it supports writing metadata to the database and object index pool, and only reading metadata in the object index pool. At this time, reading metadata in the object index pool is not affected in any way; however, the addition, deletion and modification operations of metadata in the object index pool and database will fail due to locking the target bucket.

[0155] Since header migration is a short process, a retry mechanism is added to the add, delete, and modify business processes. After the header migration is completed and the target bucket is unlocked, the retry is successful, and there is no actual impact on the user's normal business. During the process of locking the bucket and migrating the header, the user's read request is not affected in any way.

[0156] For example, in the process of migrating the header data of shards 1 to shard 9, adding, deleting, and modifying the metadata in the object index pool and the database are not allowed, but reading operations on the data in the object index pool are allowed.

[0157] Furthermore, after migrating the header, the single-write and single-read switch of the target bucket should be switched first, that is, the data addition, deletion, modification and query operations are all applied to the metadata in the database, and the data in the object index pool is no longer added, deleted, modified and queried. Then the target bucket can be unlocked and normal addition, deletion, modification and query services can be restored. From then on, the addition, deletion, modification and query services of the target bucket are all taken over by the database, and the metadata in the object index pool is no longer added, deleted, modified and queried.

[0158] Then compare_obj can be updated as follows:

[0159] {

[0160] "last_compare_time": "T0",

[0161] "current_compare_time":"T2",

[0162] "is_suspend": "false",

[0163] "interrupt_sec": 0

[0164] }

[0165] In an embodiment of the present application, in the second migration phase, the incremental metadata generated by the object index pool in the second migration phase is migrated to the database, and the migration of the header data of each shard in the target bucket is completed.

[0166] Furthermore, in the above embodiment, according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database, in the process of synchronizing the metadata of the database, that is, in the second migration phase, it is necessary to add a deletion mark to the metadata to be deleted; wherein the metadata to be deleted is the metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is the metadata shared by the database and the object index pool.

[0167] Exemplarily, in the second migration phase, adding a deletion mark to the metadata to be deleted in the database has the following reasons:

[0168] At the beginning of the second migration phase, mark-delete is enabled for the data on the database side of the bucket operation. That is, after the second migration phase begins, all deletion operations on the target bucket are still double deletions, that is, the corresponding metadata in the object index pool and the database are deleted at the same time, but a mark-delete flag is added to the metadata attribute in the database instead of deleting the metadata immediately; the corresponding metadata in the corresponding object index pool is actually deleted.

[0169] This is because the data is not locked when the metadata is compared. If 1,000 pieces of data are read from the object index pool and the metadata in the database is to be compared, and the user issues a delete operation, if it is not marked for deletion, there will be two scenarios:

[0170] Scenario 1: In the dual-write and single-read mode, that is, metadata is written to the database and object index pool, and metadata is only read from the object index pool. If the metadata is deleted successfully in both the object index pool and the database, but 1,000 metadata read from the object index pool are in the memory, the metadata will be migrated from the memory to the database, causing dirty data that should have been deleted to be present in the database. Users will be able to perceive this dirty data, which will be cleaned up in the third migration phase, so that invalid dirty data will not be generated in the end. However, during the time span from the start of the second migration phase to the end of the third migration phase, dirty data may be perceived by users.

[0171] Scenario 2: After the target bucket is switched to the single-read single-write database mode, that is, the data add, delete, modify and query operations are all applied to the metadata in the database, and the data in the object index pool is no longer added, deleted, modified and queried. All businesses access the database, so deletion is changed to deleting the metadata in the database alone, and the same metadata in the object index pool will not be deleted. If the database marked deletion is not enabled at this stage, the metadata in the database will be deleted, while the metadata in the object index pool will not be deleted. At this time, there is more metadata in the object index pool than in the database, and it is impossible to determine whether it should be migrated to the database in the third migration stage.

[0172] Therefore, if the marked deletion of metadata in the database is enabled, after switching to a single-read, single-write database, the deletion operation will mark the metadata attribute hit in the database as marked deleted, and will also update the mtime of the metadata. The metadata of the object marked for deletion will not be perceived by the user. Therefore, after the metadata comparison is completed in the third migration phase, the metadata marked for deletion in the marked deletion database can be directly cleaned up. The above two problems can be solved by marked deletion.

[0173] In an embodiment of the present application, by adding a deletion mark to the metadata to be deleted in the second migration phase, it is possible to avoid the problem of deleting data in the database without deleting data in the object index pool after the single-read single-write database mode is enabled, but the comparison result is unreliable.

[0174] In some optional implementations, the embodiment of the present application is described by taking the target migration stage as the third migration stage as an example. Assuming that the third migration stage is the time period after time T2, the first metadata to be compared is all metadata stored in the object index pool up to the end time of the second migration stage, that is, all metadata stored before time T2.

[0175] Based on this, see Fig.10 , Fig.10 A schematic diagram of another process for synchronizing metadata of a database is provided, which specifically includes the following steps:

[0176] S1001, comparing the first metadata with all metadata stored in a database to obtain a positive comparison result.

[0177] For example, the second migration phase has completed the switch to the single-read single-write database mode, but the third migration phase is still necessary. In this phase, the second metadata is the full metadata in the database. By comparing the full metadata in the object index pool and the database, the difference metadata is further processed to ensure that the metadata in the database is the latest and no dirty data remains.

[0178] At the same time, after changing the target bucket's mark deletion to actual deletion, start cleaning up the mark deletion metadata in the database. In the second migration phase, the time T2 for switching to the single-read single-write database mode has been recorded in current_compare_time and persisted. In the third migration phase, the criterion for whether inconsistent metadata should be deleted on the database side is whether the metadata's mtime is after T2.

[0179] Exemplarily, all first metadata in the object index pool may be extracted into the memory, and all metadata stored in the database may be extracted into the memory, and the first metadata may be compared with all metadata stored in the database to obtain a positive comparison result.

[0180] S1002: When the forward comparison result is that the fifth target metadata in the first metadata does not exist in the database, the fifth target metadata is synchronized to the database.

[0181] When the forward comparison result is that the fifth target metadata in the first metadata does not exist in the database, and the update time of the fifth target metadata is less than the start time of the third migration phase, which means that it is metadata that failed to be written to the database in the second migration phase, the fifth target metadata can be synchronized to the database.

[0182] For example, see Fig.11 , Fig.11A schematic diagram of the forward comparison in the third migration phase is provided. Among them, the mtime of Obj800 and Obj804 in the object index pool are both less than T2, and neither is queried in the database, so they are considered valid metadata and can be migrated to the database. Obj4 and Obj803 are metadata that exist in both the database and the object storage pool, and can be left unprocessed; Obj1, Obj3, Obj5, and Obj7 are all operations performed on the database after T2.

[0183] S1003: Compare all metadata stored in the database with the first metadata to obtain a reverse comparison result.

[0184] Furthermore, all metadata stored in the database may be compared with the first metadata to obtain a reverse comparison result.

[0185] S1004: When the reverse comparison result shows that the sixth target metadata exists in the database and the sixth target metadata does not exist in the first metadata, the sixth target metadata is deleted.

[0186] Exemplarily, when the reverse comparison result shows that the sixth target metadata exists in the database and does not exist in the first metadata, and the update time of the sixth target metadata is less than the start time of the third migration phase, the sixth target metadata is considered to be dirty data and can be deleted.

[0187] See also Fig.12 , Fig.12 A schematic diagram of the reverse comparison in the third migration phase is provided. Among them, Obj741 is dirty data and can be directly cleared; however, if the mtime is after T2, such as Obj43, no processing is performed; if the same metadata exists on both sides, such as Obj34 and Obj502, the data is considered to be consistent and no processing is performed.

[0188] S1005, deleting the metadata to be deleted that is marked for deletion in the database.

[0189] Furthermore, the target bucket can be marked for deletion and changed to actual deletion, for example, by setting the field mark_del to false. After that, all deletion operations on the target bucket are processed in the database and are actual deletion metadata, and the metadata to be deleted marked for deletion in the database is deleted.

[0190] See also Fig.13 , Fig.13A schematic diagram of cleaning up the metadata to be deleted that is marked for deletion is provided. Among them, Obj5, Obj133, Obj121 and Obj412 are the metadata to be deleted that are marked for deletion, and are actually deleted in the database; Obj41 and Obj743 are the actual deletion operations after the target bucket is marked for deletion, and Obj41 and Obj743 are actually deleted in the database.

[0191] Furthermore, compare_obj can be initialized to allow other buckets in the object index pool to migrate data.

[0192] {

[0193] "last_compare_time":"0.000000",

[0194] "current_compare_time":"0.000000",

[0195] "is_suspend": "false",

[0196] "interrupt_sec": 0

[0197] }

[0198] In an embodiment of the present application, in the third migration stage, the incremental metadata generated by the object index pool before the third migration stage is migrated to the database, and the dirty data and the metadata to be deleted marked for deletion in the database are cleared, thereby achieving the purpose of migrating the metadata in the object storage pool to the database, and there is no missing metadata or redundant dirty data in the database.

[0199] In some optional implementations, in the above embodiments, when the forward comparison result is that the first metadata exists in the second metadata, the metadata with the latest update time between the first metadata and the second metadata is synchronized to the database. For example, when the forward comparison result is that the first metadata exists in the second metadata, the update time mtime of the metadata in the object index pool and the database can be compared, and the metadata with the latest update time mtime is used as the standard, and then the metadata with the latest update time mtime is stored in the database.

[0200] In the embodiment of the present application, when the object index pool and the database both have the same metadata, the metadata with the latest update time is stored in the database, which can ensure that the metadata version stored in the database is the latest.

[0201] In some optional implementations, in order to improve data comparison efficiency, multiple concurrent threads may be used to execute the data comparison task.

[0202] Based on this, see Fig.14 , Fig.14 A flowchart of a multi-threaded data comparison task is provided, which specifically includes the following steps:

[0203] S1401, obtaining the number of threads used for comparing the first metadata and the second metadata.

[0204] Exemplarily, the number of threads used for comparing the first metadata with the second metadata, that is, the number of threads used for forward comparison, can be firstly obtained.

[0205] S1402: Determine, according to the number of shards of the target bucket in the object index pool and the number of threads, a shard allocated to each thread for comparison with the second metadata.

[0206] Furthermore, the shards allocated to each thread for comparison with the second metadata may be determined according to the number of shards of the target bucket in the object index pool and the number of threads. For example, the quotient of the number of shards of the target bucket and the number of threads may be rounded to an integer, so that the number of shards allocated to each thread is as balanced as possible.

[0207] For example, see Fig.15 , Fig.15 A schematic diagram of allocating shards to each thread is provided. Assuming there are 128 shards and 13 threads, an equal-division algorithm can be used to calculate the task list of each thread, and the shard identity (ID) is stored in the list. For example, 10 shards are allocated to the first 12 threads and 8 shards are allocated to the last thread, so that the number of shards allocated to each thread is as balanced as possible. Then, thread tasks are created, and the threads concurrently process the shard tasks in their respective task lists.

[0208] S1403, controlling each thread to perform a forward comparison operation on the first metadata and the second metadata in the corresponding slice to obtain a forward comparison result.

[0209] Furthermore, each thread is controlled to perform a forward comparison operation on the first metadata and the second metadata in the respective allocated slices to obtain a forward comparison result, thereby improving the efficiency of the forward comparison.

[0210] S1404: Determine, according to the number of second metadata and the number of threads, second metadata allocated to each thread for comparison with the first metadata.

[0211] Exemplarily, the number of threads used to compare the second metadata with the first metadata is the number of threads used for reverse comparison. Similarly, the second metadata allocated to each thread for comparison with the first metadata can be determined based on the number of second metadata and the number of threads. For example, the quotient of the number of second metadata and the number of threads can be rounded to an integer, so that the number of second metadata allocated to each thread is as balanced as possible.

[0212] For example, see Fig.16 , Fig.16 A schematic diagram of allocating second metadata to each thread is provided. Since there is no concept of sharding in the database, it is impossible to split the subtask list by sharding. Therefore, the subtask is split by [start_key, end_key] and assembled into multiple groups of [start_key, end_key] combinations, and the multiple groups of lists are given to the corresponding thread tasks. Among them, start_key and end_key are the split interval index values, and the corresponding second metadata can be queried according to the interval index value.

[0213] The first group start_key is empty, indicating that the query starts from the interval index value 0, and the last group end_key is empty, indicating that the query ends at the end. Inside the thread task, the data is enumerated according to the given start_key and end_key. If the last key enumerated is greater than or equal to the end_key in lexicographic order, it means that the thread task can be terminated. In this way, each concurrent thread can process the same second metadata entry as much as possible.

[0214] Assuming that there are 10,000 pieces of second metadata and 10 threads, an equal distribution algorithm can be used to allocate 1,000 pieces of second metadata to each thread, so as to make the number of second metadata allocated to each thread balanced as much as possible.

[0215] S1405, controlling each thread to perform a reverse comparison operation on the corresponding second metadata and the first metadata to obtain a reverse comparison result.

[0216] Furthermore, each thread can be controlled to concurrently perform a reverse comparison operation on the corresponding second metadata and the first metadata to obtain a reverse comparison result, thereby improving the efficiency of the reverse comparison.

[0217] In the embodiment of the present application, by using multiple concurrent threads to execute the forward comparison task and the reverse comparison task, the efficiency of the forward comparison and the reverse comparison is improved while ensuring that the tasks assigned to each thread are balanced.

[0218] In some optional implementations, see Fig.17 , Fig.17Another data synchronization process diagram is provided, which specifically includes the following steps:

[0219] S1701, obtaining first metadata corresponding to each shard of a target bucket in an object index pool during a target migration phase, and second metadata corresponding to a database during a target migration phase.

[0220] S1702: Obtain the number of threads used to compare the first metadata and the second metadata.

[0221] S1703: Determine, according to the number of shards of the target bucket in the object index pool and the number of threads, a shard allocated to each thread for comparison with the second metadata.

[0222] S1704, controlling each thread to perform a forward comparison operation on the first metadata and the second metadata in the corresponding slice to obtain a forward comparison result.

[0223] S1705: Determine, according to the number of second metadata and the number of threads, second metadata allocated to each thread for comparison with the first metadata.

[0224] S1706, controlling each thread to perform a reverse comparison operation on the corresponding second metadata and the first metadata to obtain a reverse comparison result.

[0225] S1707: Synchronize metadata on the database according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database.

[0226] In the process of uploading a large amount of file data, the embodiment of the present application no longer relies on the concept of sharding metadata storage, and avoids the business bottleneck of large omap alarms and significantly reduced enumeration efficiency when the number of objects in a single bucket increases from a design perspective, and the response efficiency to services such as life cycle will also be faster. In addition, the embodiment of the present application adopts the current phased bidirectional multi-concurrency comparison migration solution, which not only ensures metadata consistency, but also greatly improves metadata comparison migration efficiency, preventing the potential impact of a long migration process on the original business of the cluster.

[0227] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0228] Based on the same inventive concept, the embodiment of the present application also provides a data synchronization device for implementing the data synchronization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data synchronization device embodiments provided below can refer to the limitations on the data synchronization method above, and will not be repeated here.

[0229] In one embodiment, Fig.18 As shown, a data synchronization device is provided, comprising:

[0230] The acquisition module 10 is used to obtain the first metadata corresponding to each shard of the target bucket in the object index pool in the target migration stage; wherein the target migration stage is the first migration stage, the second migration stage or the third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage;

[0231] The synchronization module 20 is used to synchronize metadata of the database according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0232] The above-mentioned data synchronization device obtains the first metadata corresponding to each shard of the target bucket in the object index pool in the target migration stage; wherein the target migration stage is the first migration stage, the second migration stage or the third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage; according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database, the database is synchronized with the metadata; wherein the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata. The above-mentioned scheme compares the first metadata in the object index pool and the second metadata in the database in stages, and synchronizes the first metadata to the database, that is, the database is used to store metadata, and bucket storage is not involved, thereby solving the problem caused by too many or too few shards in the bucket; and, through the forward comparison and the reverse comparison, it can be ensured that the first metadata in the object index pool is stored in the database, and there will be no dirty data in the database.

[0233] In one embodiment, when the target migration phase is the first migration phase, the first metadata is metadata stored in the object index pool before the start time of the first migration phase; the synchronization module 20 is specifically used to:

[0234] The first metadata is compared with the metadata stored in the database before the start time of the first migration phase to obtain a forward comparison result; when the forward comparison result is that the first target metadata in the first metadata does not exist in the database, the first target metadata is synchronized to the database; all metadata stored in the database are compared with the first metadata to obtain a reverse comparison result; when the reverse comparison result is that the second target metadata exists in the database and the second target metadata does not exist in the first metadata, the second target metadata is deleted from the database.

[0235] In one embodiment, when the target migration phase is the second migration phase, the first metadata is metadata stored in the object index pool between the start time of the first migration phase and the end time of the first migration phase; the synchronization module 20 is specifically used to:

[0236] Compare the first metadata with the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a forward comparison result;

[0237] If the forward comparison result is that the third target metadata in the first metadata does not exist in the database, the third target metadata is synchronized to the database; the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase is compared with the first metadata to obtain a reverse comparison result; if the reverse comparison result is that the fourth target metadata exists in the database and the fourth target metadata does not exist in the first metadata, the fourth target metadata is deleted from the database; after locking the metadata for non-read operations in the target bucket, the header data of each shard of the target bucket is migrated to the database.

[0238] In one of the embodiments, the device further comprises an adding module, configured to:

[0239] In a process of synchronizing metadata of a database according to a forward comparison result and a reverse comparison result between first metadata and second metadata stored in the database, in a second migration phase, a deletion mark is added to metadata to be deleted; wherein the metadata to be deleted is metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is metadata shared by the database and the object index pool.

[0240] In one embodiment, when the target migration phase is the third migration phase, the first metadata is all metadata stored in the object index pool up to the end time of the second migration phase; the synchronization module 20 is specifically used for:

[0241] The first metadata is compared with all metadata stored in the database to obtain a forward comparison result; when the forward comparison result is that the fifth target metadata in the first metadata does not exist in the database, the fifth target metadata is synchronized to the database; wherein the update time of the fifth target metadata is less than the start time of the third migration stage; all metadata stored in the database are compared with the first metadata to obtain a reverse comparison result; when the reverse comparison result is that the sixth target metadata exists in the database and does not exist in the first metadata, the sixth target metadata is deleted; wherein the update time of the sixth target metadata is less than the start time of the third migration stage; and the metadata to be deleted marked for deletion in the database is deleted.

[0242] In one embodiment, the synchronization module 20 is further configured to:

[0243] When the forward comparison result is that the first metadata exists in the second metadata, the metadata with the latest update time between the first metadata and the second metadata is synchronized to the database.

[0244] In one embodiment, the device further comprises a comparison module, which is used to:

[0245] The number of threads used for comparing the first metadata with the second metadata is obtained; according to the number of shards of the target bucket in the object index pool and the number of threads, the shards allocated to each thread for comparison with the second metadata are determined; each thread is controlled to perform a forward comparison operation between the first metadata and the second metadata in the corresponding shard to obtain a forward comparison result; according to the number of the second metadata and the number of threads, the second metadata allocated to each thread for comparison with the first metadata is determined; each thread is controlled to perform a reverse comparison operation between the corresponding second metadata and the first metadata to obtain a reverse comparison result.

[0246] Each module in the above data synchronization device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0247] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.19 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store object data and object metadata. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0248] Those skilled in the art will understand that Fig.19 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0249] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the data synchronization method described in any of the above embodiments are implemented.

[0250] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data synchronization method described in any of the above embodiments are implemented.

[0251] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the data synchronization method described in any of the above embodiments when executed by a processor.

[0252] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0253] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0254] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0255] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Obtaining first metadata corresponding to each shard of a target bucket in an object index pool in a target migration phase; wherein the target migration phase is a first migration phase, a second migration phase, or a third migration phase, the end time of the first migration phase is the start time of the second migration phase, and the end time of the second migration phase is the start time of the third migration phase; Perform metadata synchronization on the database according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata; In the case where the target migration phase is the first migration phase, the first metadata is metadata stored in the object index pool before the start time of the first migration phase; the second metadata is metadata stored in the database before the start time of the first migration phase; The step of synchronizing the database with metadata according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database includes: Comparing the first metadata with metadata stored in the database before the start time of the first migration phase to obtain a positive comparison result; When the forward comparison result is that the first target metadata in the first metadata does not exist in the database, synchronizing the first target metadata to the database; Comparing the second metadata with the first metadata to obtain a reverse comparison result; When the reverse comparison result is that the second target metadata exists in the database and the second target metadata does not exist in the first metadata, the second target metadata is deleted from the database.

2. The method according to claim 1, characterized in that In a case where the target migration phase is the second migration phase, the first metadata is metadata stored in the object index pool between the start time of the first migration phase and the end time of the first migration phase; The second metadata is metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase; The step of synchronizing the database with metadata according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database includes: Comparing the first metadata with metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a forward comparison result; When the forward comparison result is that the third target metadata in the first metadata does not exist in the database, synchronizing the third target metadata to the database; Comparing the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase with the first metadata to obtain a reverse comparison result; When the reverse comparison result shows that the fourth target metadata exists in the database and the fourth target metadata does not exist in the first metadata, deleting the fourth target metadata from the database; After the metadata for non-read operations in the target bucket is locked, the header data of each shard of the target bucket is migrated to the database.

3. The method according to claim 2, characterized in that In the process of synchronizing the metadata of the database according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database, the method further includes: In the second migration phase, a deletion mark is added to the metadata to be deleted; The metadata to be deleted is metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is metadata common to the database and the object index pool.

4. The method according to claim 1, characterized in that: In the case where the target migration stage is the third migration stage, the first metadata is all metadata stored in the object index pool up to the end time of the second migration stage; the second metadata is all metadata stored in the database; The step of synchronizing the database with metadata according to the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database includes: Comparing the first metadata with all metadata stored in the database to obtain a positive comparison result; When the forward comparison result is that the fifth target metadata in the first metadata does not exist in the database, synchronizing the fifth target metadata to the database; wherein the update time of the fifth target metadata is less than the start time of the third migration stage; Comparing all metadata stored in the database with the first metadata to obtain a reverse comparison result; If the reverse comparison result shows that the sixth target metadata exists in the database and the sixth target metadata does not exist in the first metadata, the sixth target metadata is deleted; wherein the update time of the sixth target metadata is less than the start time of the third migration stage; The metadata to be deleted that is marked for deletion in the database is deleted.

5. The method according to claim 1, characterized in that The method further comprises: When the forward comparison result is that the first metadata exists in the second metadata, the metadata with the latest update time between the first metadata and the second metadata is synchronized to the database.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining the number of threads used for comparing the first metadata with the second metadata; Determine, according to the number of shards of the target bucket in the object index pool and the number of threads, a shard allocated to each thread for comparison with the second metadata; Control each thread to perform a forward comparison operation on the first metadata in the corresponding shard and the second metadata to obtain a forward comparison result; Determining, according to the number of the second metadata and the number of threads, second metadata allocated to each thread for comparison with the first metadata; Each thread is controlled to perform a reverse comparison operation on the corresponding second metadata and the first metadata to obtain a reverse comparison result.

7. A data synchronization device, characterized in that: The device comprises: An acquisition module, used to acquire first metadata corresponding to each shard of a target bucket in an object index pool in a target migration phase; wherein the target migration phase is a first migration phase, a second migration phase, or a third migration phase, the end time of the first migration phase is the start time of the second migration phase, and the end time of the second migration phase is the start time of the third migration phase; a synchronization module, configured to synchronize metadata on the database according to a forward comparison result and a reverse comparison result between the first metadata and the second metadata stored in the database; wherein the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata; When the target migration phase is the first migration phase, the first metadata is metadata stored in the object index pool before the start time of the first migration phase; the second metadata is metadata stored in the database before the start time of the first migration phase; the synchronization module is specifically used for: The first metadata is compared with the metadata stored in the database before the start time of the first migration phase to obtain a forward comparison result; if the forward comparison result is that the first target metadata in the first metadata does not exist in the database, the first target metadata is synchronized to the database; the second metadata is compared with the first metadata to obtain a reverse comparison result; if the reverse comparison result is that the second target metadata exists in the database and the second target metadata does not exist in the first metadata, the second target metadata is deleted from the database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images