Data synchronization method and apparatus, computer device, and storage medium

Abstract

The application relates to a data synchronization method and device, computer equipment and a storage medium. First metadata corresponding to each shard of a target bucket in an object index pool in a target migration stage is acquired; metadata synchronization is performed on a database according to a forward comparison result and a reverse comparison result between the first metadata and second metadata stored in the database; the forward comparison result is a result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is a result of comparing whether the second metadata exists in the first metadata. According to the above scheme, the first metadata in the object index pool and the second metadata in the database are compared in stages, and the first metadata is synchronized to the database, that is, the database stores the metadata, and no bucket storage is involved, thereby solving the problem caused by too many or too few shards in the bucket; and the first metadata in the object index pool is ensured to be stored in the database, and no dirty data exists.

Description

[0001] This application is a divisional application of application number 2025100072652 (Data synchronization method, apparatus, computer equipment and storage medium, application date: January 3, 2025). Technical Field

[0002] This application relates to the field of data storage technology, and in particular to a data synchronization method, apparatus, computer equipment, and storage medium. Background Technology

[0003] Metadata can be understood as data that describes data attributes, and it has functions such as recording storage location, recording historical data, and data retrieval. With the rapid development of computer technology, various types of data are experiencing explosive growth, and managing and storing metadata for these types of data also faces new challenges.

[0004] In traditional technologies, a new bucket is typically created to store object data. This bucket is then divided into a specified number of shards to store the object data's associated metadata. For example, when object data is uploaded, a consistent hashing algorithm is used to calculate the target shard for the object data's associated metadata, and this target shard is then used to store the associated metadata.

[0005] However, too few shards in a bucket can lead to insufficient concurrent data storage capacity; too many shards can lead to high latency in enumerating object data; and dynamically adjusting the number of shards can cause business interruptions. Summary of the Invention

[0006] Therefore, it is necessary to provide a data synchronization method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems, which can solve the problems caused by too many or too few shards in a bucket, as well as the problems caused by dynamic adjustment of the number of shards.

[0007] Firstly, this application provides a data synchronization method, the method comprising:

[0008] Obtain the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein, the target migration stage is a first migration stage, a second migration stage, or a third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage.

[0009] Based on the forward comparison results and reverse comparison results between the first metadata and the second metadata stored in the database, the database is synchronized with metadata; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0010] In one embodiment, when the target migration phase is a first migration phase, the first metadata is metadata stored in the object index pool before the start time of the first migration phase;

[0011] The step of synchronizing the database based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database includes:

[0012] The first metadata is compared with the metadata stored in the database before the start of the first migration phase to obtain a positive comparison result;

[0013] If the forward comparison result indicates that the first target metadata in the first metadata does not exist in the database, the first target metadata will be synchronized to the database.

[0014] The first metadata is compared with all the metadata stored in the database to obtain the reverse comparison result;

[0015] If the reverse comparison result indicates that the second target metadata exists in the database, but the second target metadata does not exist in the first metadata, the second target metadata is deleted from the database.

[0016] In one embodiment, when the target migration stage is a second migration stage, the first metadata is the metadata stored in the object index pool between the start time of the first migration stage and the end time of the first migration stage;

[0017] The step of synchronizing the database based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database includes:

[0018] The first metadata is compared with the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a positive comparison result;

[0019] If the forward comparison result indicates that the third target metadata in the first metadata does not exist in the database, the third target metadata will be synchronized to the database.

[0020] The metadata stored in the database between the start time and the end time of the first migration phase is compared with the first metadata to obtain a reverse comparison result;

[0021] If the reverse comparison result indicates that the fourth target metadata exists in the database, but the fourth target metadata does not exist in the first metadata, then the fourth target metadata is deleted from the database.

[0022] After locking the metadata of the target bucket that is not being read, the header data of each shard of the target bucket is migrated to the database.

[0023] In one embodiment, during the process of synchronizing the database based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database, the method further includes:

[0024] In the second migration phase, deletion tags are added to the metadata to be deleted;

[0025] The metadata to be deleted refers to the metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is metadata shared by the database and the object index pool.

[0026] In one embodiment, when the target migration stage is the third migration stage, the first metadata is all metadata stored in the object index pool up to the end of the second migration stage.

[0027] The step of synchronizing the database based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database includes:

[0028] The first metadata is compared with all metadata stored in the database to obtain a positive comparison result;

[0029] If the forward comparison result indicates that the fifth target metadata in the first metadata does not exist in the database, the fifth target metadata is synchronized to the database; wherein the update time of the fifth target metadata is less than the start time of the third migration phase;

[0030] The first metadata is compared with all the metadata stored in the database to obtain the reverse comparison result;

[0031] If the reverse comparison result indicates that the sixth target metadata exists in the database, but the sixth target metadata does not exist in the first metadata, the sixth target metadata is deleted; wherein the update time of the sixth target metadata is less than the start time of the third migration phase;

[0032] Delete the metadata in the database that is marked for deletion.

[0033] In one embodiment, the method further includes:

[0034] If the forward comparison result shows that the first metadata exists in the second metadata, the first metadata and the metadata with the latest update time in the second metadata are synchronized to the database.

[0035] In one embodiment, the method further includes:

[0036] Obtain the number of threads used to compare the first metadata and the second metadata;

[0037] Based on the number of shards in the target bucket of the object index pool and the number of threads, determine the shards allocated to each thread for comparison with the second metadata.

[0038] Control each thread to perform a forward comparison operation between the first metadata in the corresponding slice and the second metadata, and obtain the forward comparison result;

[0039] Based on the quantity of the second metadata and the quantity of threads, determine the second metadata allocated to each thread for comparison with the first metadata;

[0040] Control each thread to perform a reverse comparison operation between the corresponding second metadata and the first metadata, and obtain the reverse comparison result.

[0041] Secondly, this application also provides a data synchronization device, the device comprising:

[0042] The acquisition module is used to acquire the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein the target migration stage is a first migration stage, a second migration stage, or a third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage.

[0043] The synchronization module is used to synchronize the database metadata based on the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; and the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0044] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0045] Obtain the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein, the target migration stage is a first migration stage, a second migration stage, or a third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage.

[0046] Based on the forward comparison results and reverse comparison results between the first metadata and the second metadata stored in the database, the database is synchronized with metadata; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0048] Obtain the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein, the target migration stage is a first migration stage, a second migration stage, or a third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage.

[0049] Based on the forward comparison results and reverse comparison results between the first metadata and the second metadata stored in the database, the database is synchronized with metadata; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0050] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0051] Obtain the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein, the target migration stage is a first migration stage, a second migration stage, or a third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage.

[0052] Based on the forward comparison results and reverse comparison results between the first metadata and the second metadata stored in the database, the database is synchronized with metadata; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0053] The aforementioned data synchronization method, apparatus, computer equipment, and storage medium acquire the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage. The target migration stage can be a first migration stage, a second migration stage, or a third migration stage. The end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage. Based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database, metadata synchronization is performed on the database. The forward comparison result checks whether the first metadata exists within the second metadata; the reverse comparison result checks whether the second metadata exists within the first metadata. This scheme, by comparing the first metadata in the object index pool with the second metadata in the database in stages and synchronizing the first metadata to the database, uses database storage for metadata, avoiding bucket storage, thus solving the problem of too many or too few shards within a bucket. Furthermore, the forward and reverse comparisons ensure that all first metadata in the object index pool is stored in the database, and that there is no dirty data in the database. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a data synchronization method in one embodiment;

[0055] Figure 2 This is a schematic diagram illustrating the storage of metadata in an object index pool in one embodiment.

[0056] Figure 3 This is a schematic diagram illustrating the process of synchronizing metadata in a database in one embodiment;

[0057] Figure 4 This is a schematic diagram of the forward alignment in the first migration stage of one embodiment;

[0058] Figure 5 This is a schematic diagram of the reverse alignment in the first migration stage of one embodiment;

[0059] Figure 6 This is a schematic diagram of another process for synchronizing metadata in a database, as shown in one embodiment.

[0060] Figure 7 This is a schematic diagram of the forward alignment in the second migration stage in one embodiment;

[0061] Figure 8 This is a schematic diagram of the reverse alignment in the second migration stage of one embodiment;

[0062] Figure 9 This is a schematic diagram of locking and migrating header data in one embodiment;

[0063] Figure 10 This is a schematic diagram illustrating another process for synchronizing metadata in a database, as shown in one embodiment.

[0064] Figure 11 This is a schematic diagram of the forward alignment in the third migration stage of one embodiment;

[0065] Figure 12 This is a schematic diagram of the reverse alignment in the third migration stage of one embodiment;

[0066] Figure 13 This is a schematic diagram illustrating the cleaning of metadata marked for deletion in one embodiment;

[0067] Figure 14 This is a flowchart illustrating the process of performing a data comparison task using multiple threads in one embodiment.

[0068] Figure 15 This is a schematic diagram illustrating the allocation of fragments to each thread in one embodiment;

[0069] Figure 16 This is a schematic diagram illustrating the allocation of second metadata to each thread in one embodiment;

[0070] Figure 17 This is a schematic diagram of another data synchronization process in one embodiment;

[0071] Figure 18 This is a structural block diagram of a data synchronization device in one embodiment;

[0072] Figure 19 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0074] The data synchronization method provided in this application can be applied to scenarios where metadata stored in an object index pool is migrated to a distributed database. This method can be executed by a server or by a terminal with sufficient computing power.

[0075] The server can be a standalone server or a server cluster consisting of multiple servers. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc.

[0076] Against the backdrop of rapid development in the internet, the Internet of Things, cloud computing, and big data, data is experiencing explosive growth. Data generated by various applications such as social networks, mobile communications, online video and audio, e-commerce, sensor networks, and scientific experiments not only has large storage capacity but also exhibits significant characteristics such as diverse data types, large variations in data size, and rapid flow, often resulting in massive amounts of files, even reaching hundreds of millions or billions of records. This presents enormous challenges in metadata management. Object storage services currently face the following three problems in managing object metadata within their storage processes.

[0077] 1. When a new bucket is created to store data, the bucket is divided into a specified number of shards to store object-related metadata information. This information is stored using an object map and key-value pair (OMAP(kv)) structure. Each shard corresponds to a Reliable Autonomic Distributed ObjectStore (rados) object. A rados object consists of three parts: an object identifier, object data, and object metadata. The object identifier uniquely identifies an object; the object data corresponds to a file in the local file system, where the object data is stored; and the object metadata is stored in key-value pairs.

[0078] When uploading a file object, a consistent hashing algorithm can be used to calculate the target shard for the object's metadata and store the file object's metadata information. The consistent hashing algorithm ensures that each shard can carry the file object's metadata as evenly as possible. However, as the number of objects in the bucket increases, the number of metadata entries stored on each shard will also increase. When the alarm threshold limited by the cluster is reached, a large number of object metadata mapping (large omap) alarms will occur. At this time, a large number of concurrent requests to the bucket will increase the number of requests falling on the same shard. Since the consistency of Rados object data is guaranteed by a single thread, it will directly affect the execution efficiency of user operations such as uploading, downloading, and deleting object data.

[0079] 2. The cluster supports resharding existing buckets, which increases the number of shards in a bucket and simultaneously rebalances the data, reducing the number of metadata (omap) entries stored on a single shard. When concurrent requests arrive, because the data is distributed across more shards, request blocking on a single shard is further reduced, thus improving concurrency performance. However, during bucket resharding, data migration may occur, potentially causing partial service interruptions.

[0080] 3. Increasing the number of shards when creating a new bucket will also introduce new problems. When the number of files in the bucket reaches hundreds of millions, the number of OMAP entries stored on each shard increases accordingly. For a single shard, the metadata key-value pairs (omap_key) of the objects it manages are ordered, but the data between shards is not ordered. Therefore, when a request to enumerate 1000 objects in batches is issued, the processing logic is to retrieve a certain number of entries from each shard, then merge and sort all the enumerated data, and select the top 1000 entries in lexicographical order as the response result. Therefore, when the number of shards is too large, the enumeration performance will be significantly reduced.

[0081] Based on this, this application introduces the Advanced Distributed SQL Database (TiDB) to manage object metadata, that is, migrating the metadata in the object index pool to the TiDB database. TiDB is an open-source distributed relational database that supports both online transaction processing and online analytical processing. It has the following advantages:

[0082] 1. TiDB is a distributed storage database that enables replica-level disaster recovery backup, with a default of 3 replicas.

[0083] 2. TiDB database supports massive data and high-concurrency Online Transaction Processing (OLTP) scenarios, eliminating the need to consider the problem of performance degradation caused by sharding resource contention and competition.

[0084] 3. TiDB's database features a storage-compute separation architecture, allowing for online scaling up or down of both compute and storage as needed, with the scaling process transparent to application maintenance personnel.

[0085] 4. TiDB database key-value pairs are globally strictly ordered. Based on this characteristic, the concept of sharding can be completely eliminated, and the performance of data enumeration will be significantly improved.

[0086] The implementation steps of the data synchronization method provided in the embodiments of this application will be described in detail below.

[0087] In one embodiment, such as Figure 1 As shown, a data synchronization method is provided. Taking the application of this method to a server as an example, the method includes the following steps:

[0088] S101, obtain the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration phase.

[0089] For example, see Figure 2 , Figure 2This document provides a schematic diagram of an object index pool storing metadata. The object index pool comprises multiple buckets, each of which can be divided into multiple shards. The target bucket is any bucket within the object index pool. Metadata is stored using an omap(kv) structure. Taking shard 1 and shard 2 as examples, omap_key is the metadata key-value pair, and type is the type of omap_key. Shard 1 stores three single-version object metadata: single_obj1, single_obj2, and single_obj3; all single-version object metadata are of plain type. Shard 2 stores two versions of a file. ver_obj is of plain type and has no practical use, existing as a placeholder. ver_obj_1 and ver_obj_2 are also of plain type, representing placeholders for these two versions. When listing all versions of this object, ver_obj_1 and ver_obj_2 will be listed. The types `ver_obj_1_1000` and `ver_obj_2_1000` are of type `instance`, representing the instance metadata identifier information corresponding to a specific version. That is, the specific instance corresponding to `ver_obj_1` is `ver_obj_1_1000`. The last one, `ver_obj_1_1001`, is of type `latest version` (olh); it points to the latest version information of the object. When no version number is specified, the latest version information of the object will be found based on the olh information. The statistics for each shard are stored in the header data of that shard. For example, the statistics for shard 1 are stored in the `shard1_header` attribute of the shard header; the statistics for shard 2 are stored in the `shard2_header` attribute of the shard header. The header records statistical information such as the number of objects stored on the shard and its capacity. The number of objects in a bucket and the amount of capacity occupied are statistically summarized from the collected shard header information.

[0090] The target migration stage can be designated as the first migration stage, the second migration stage, or the third migration stage. The end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage. For example, assuming time T0 is the start time of the first migration stage and T1 is the end time of the first migration stage, the time period between time T0 and T1 is the first migration stage; the start time of the second migration stage is time T1, and assuming the end time of the second migration stage is time T2, the time period between time T1 and T2 is the second migration stage; the start time of the third migration stage is time T2, and the time period after time T2 is the third migration stage.

[0091] Optionally, a comparison object `compare_obj` can be introduced, which internally records the start comparison time `last_compare_time` and the end comparison time `current_compare_time` as the two ends of the migration phase, with the unit being microseconds (μs). Additionally, a pause flag `is_suspend(bool)` and a pause duration flag `interrupt_sec(s)` are added to record whether the task is paused and the pause duration (in seconds), respectively. When the comparison task puts pressure on the cluster and increases the latency of normal business operations, the comparison task can be paused at any time and resumed when the cluster's business pressure decreases.

[0092] For example, the initialized compare_obj is as follows:

[0093]

[0094] For example, the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage is stored on disk. The first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage can be read into memory for data comparison operations.

[0095] S102, Synchronize the metadata of the database based on the forward comparison results and the reverse comparison results between the first metadata and the second metadata stored in the database.

[0096] The forward alignment result is the result of comparing whether the first metadata exists in the second metadata; the reverse alignment result is the result of comparing whether the second metadata exists in the first metadata.

[0097] For example, metadata synchronization can be performed on the database first based on the forward comparison results. For instance, first metadata not included in the second metadata of the database can be synchronized to the database. Then, metadata synchronization can be performed on the database based on the reverse comparison results. For instance, second metadata not present in the first metadata of the object index pool can be considered dirty data and deleted from the database. The database can be a TiDB database.

[0098] The aforementioned data synchronization method obtains the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage. The target migration stage can be a first migration stage, a second migration stage, or a third migration stage. The end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage. Based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database, metadata synchronization is performed on the database. The forward comparison result checks whether the first metadata exists within the second metadata; the reverse comparison result checks whether the second metadata exists within the first metadata. This scheme, by comparing the first metadata in the object index pool with the second metadata in the database in stages and synchronizing the first metadata to the database, uses database storage for metadata, avoiding bucket storage, thus solving the problem of too many or too few shards within a bucket. Furthermore, the forward and reverse comparisons ensure that all first metadata in the object index pool is stored in the database, and that there is no dirty data in the database.

[0099] In some optional implementations, this application embodiment uses the target migration stage as the first migration stage as an example for illustration. Assuming that the first migration stage is the time period between time T0 and time T1, the first metadata to be compared is the metadata stored in the object index pool before the start time of the first migration stage, that is, the metadata before the start time T0.

[0100] Based on this, see Figure 3 , Figure 3 A flowchart illustrating the process of synchronizing metadata in a database is provided, which includes the following steps:

[0101] S301, compare the first metadata with the metadata stored in the database before the start of the first migration phase to obtain a positive comparison result.

[0102] For example, compare_obj at the start of the first migration phase is as follows:

[0103]

[0104] At this point, the second metadata is the metadata stored in the database before the start of the first migration phase. The metadata stored in the database before the start of the first migration phase can be read into memory, and the first metadata can be compared with the metadata stored in the database before the start of the first migration phase to obtain a positive comparison result.

[0105] S302, if the forward comparison result shows that the first target metadata does not exist in the first metadata in the database, synchronize the first target metadata to the database.

[0106] Furthermore, if the forward comparison result indicates that the first target metadata in the first metadata does not exist in the database, the full data of the first target metadata is copied to the database. If the forward comparison result indicates that the first target metadata in the first metadata exists in the database, the update time mtime (recorded in the metadata attribute field) of that metadata in the object index pool and the database can be compared. The metadata with the latest update time mtime is used as the standard, and then the metadata with the latest update time mtime is stored in the database.

[0107] For example, see Figure 4 , Figure 4 A schematic diagram of the forward comparison in the first migration phase is provided. In this diagram, the metadata key values ​​Obj1, Obj4, and Obj801 in shard 1 have update times less than T0, and the metadata key values ​​Obj3, Obj6, and Obj800 in shard 2 have update times less than T0; both can be used as first metadata. If the forward comparison result indicates that these data do not exist in the database, they are synchronized to the database. Obj2 and Obj5 are deleted data and do not need to be synchronized to the database. Obj7 and Obj8 have update times greater than T0 and are modified data; therefore, Obj7 and Obj8 are not processed in the first migration phase. Obj804 and Obj802 have update times greater than T0 and are written data. Since the first migration phase supports simultaneous writing of data to the object index pool and the database, the metadata corresponding to Obj804 and Obj802 will be written to the database; however, data is only read from the object index pool.

[0108] S303, compare all metadata stored in the database with the first metadata to obtain the reverse comparison result.

[0109] Furthermore, all metadata stored in the database can be extracted into memory, and then compared with the first metadata to obtain a reverse comparison result.

[0110] S304. If the reverse comparison result shows that the second target metadata exists in the database, but the first metadata does not contain the second target metadata, the second target metadata is deleted from the database.

[0111] Furthermore, if the reverse comparison result shows that the second target metadata exists in the database, but the first metadata does not contain the second target metadata, the second target metadata can be considered dirty data and deleted from the database.

[0112] For example, see Figure 5 , Figure 5A schematic diagram of the reverse comparison in the first migration stage is provided. Among them, Obj34, Obj41, and Obj502, whose update time is less than T0 and exist in the first metadata, are not processed. Obj43, whose update time is greater than T0, is not processed in the first migration stage. Obj107 and Obj504, whose update time is less than T0 and do not exist in the first metadata, are considered dirty data and can be deleted from the database.

[0113] Furthermore, after all secondary metadata in the database has been traversed, `last_compare_time` (T0) is assigned to `current_compare_time` and persisted. This means that the data compared in the first migration phase was updated before T0. New data generated during the comparison process will have an update time greater than T0, and this new data will be processed in the second migration phase. Thus, the entire comparison process for the first migration phase is complete.

[0114] Therefore, compare_obj can be updated as follows:

[0115]

[0116] In this embodiment of the application, during the first migration phase, all the first metadata stored in the object index pool before the start of the first migration phase is migrated to the database, thus completing the migration of the existing data before the start of the first migration phase.

[0117] In some optional implementations, this application embodiment uses the second migration stage as an example for illustration. Assuming the second migration stage is the time period between time T1 and time T2, the first metadata to be compared is the metadata stored in the object index pool between the start time and the end time of the first migration stage, that is, the metadata between time T1 and time T2.

[0118] Based on this, see Figure 6 , Figure 6 An alternative flowchart for synchronizing metadata in a database is provided, which includes the following steps:

[0119] S601, compare the first metadata with the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a positive comparison result.

[0120] For example, compare_obj at the start of the second migration phase is as follows:

[0121]

[0122] At the start of the second migration phase, simultaneous writing of data to both the object index pool and the database is still supported, while data retrieval is only performed in the object index pool. Furthermore, the target bucket mark-deletion switch is enabled, which adds a mark_del field to the target bucket's attribute data. The default value is false; changing it to true and persisting it to the underlying layer will then delete the actual data for objects within that target bucket. However, the mark_del attribute in the metadata of the TiDB database will be marked as true, meaning the metadata in the TiDB database is only marked for deletion, not actually deleted, while the metadata in the object index pool will be truly deleted.

[0123] At this point, the second metadata refers to the metadata stored in the database between the start and end times of the first migration phase. The metadata stored in the object index pool between the start and end times of the first migration phase can be extracted into memory, as can the metadata stored in the database between the start and end times of the first migration phase. The first metadata is then compared with the metadata stored in the database between the start and end times of the first migration phase to obtain a positive comparison result.

[0124] S602, if the forward comparison result shows that the third target metadata does not exist in the first metadata in the database, the third target metadata is synchronized to the database.

[0125] For example, if the forward comparison result indicates that the third target metadata in the first metadata does not exist in the database, the third target metadata can be synchronized to the database. If the forward comparison result indicates that the third target metadata in the first metadata exists in the database, the update time mtime of that metadata in the object index pool and the database can be compared. The metadata with the latest update time mtime can be used as the standard, and then the metadata with the latest update time mtime can be stored in the database.

[0126] For example, see Figure 7 , Figure 7This diagram illustrates the forward comparison in the second migration phase. Obj5 represents existing data prior to T0, which was deleted after T1. Therefore, the deletion was actually completed in the object index pool, but the database was marked for deletion and its mtime was updated to be greater than T1, meaning it wasn't actually deleted. Obj7 represents updated existing data; therefore, the database and object index pool were updated during the comparison process, and the update time mtime is greater than T1. Thus, Obj7 is not part of the comparison metadata in this phase. Obj800 represents newly added business data. In this case, a dual-write mode is used, so a new object metadata record is added to both the database and the object index pool. Therefore, only Obj4, which was uploaded after the start of the first migration phase (after T0) and failed to write to the database, can be migrated from the object index pool to the database. Obj1, Obj3, Obj803, and Obj804 are all existing data already present in the database.

[0127] S603, compare the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase with the first metadata to obtain the reverse comparison result.

[0128] Furthermore, the metadata stored in the database between the start and end times of the first migration phase can be compared with the first metadata to obtain the reverse comparison result.

[0129] S604. If the reverse comparison result shows that the fourth target metadata exists in the database, but the fourth target metadata does not exist in the first metadata, the fourth target metadata is deleted from the database.

[0130] Furthermore, if the reverse comparison result shows that the fourth target metadata exists in the database, but the fourth target metadata does not exist in the first metadata, the fourth target metadata is considered to be dirty data and can be deleted from the database.

[0131] See Figure 8 , Figure 8 A schematic diagram of the reverse comparison in the second migration phase is provided. In this diagram, if the metadata Obj41 was deleted after T1, the database marks Obj41 as deleted and updates the time to be greater than T1, while Obj41 in the object index pool is actually deleted. New metadata Obj43 is added simultaneously to both the database and the object index pool; metadata that exists in the database but not in the object index pool (Obj107, Obj504), i.e., the fourth target metadata, is considered dirty data and is removed from the database; updating metadata Obj741 is also done simultaneously in both the database and the object index pool. Obj34 and Obj502 are metadata that exist in both the database and the object index pool and do not require processing.

[0132] S605: After locking the metadata of the object index pool that is not being read, migrate the header data of each shard in the target bucket of the object index pool to the database.

[0133] For example, since the metadata is not locked, a phased comparison migration method is adopted during the metadata migration process to consider the final data consistency. However, when migrating the header of a shard, the target bucket must be locked so that all add, delete, and modify operations on the target bucket are not allowed before the header migration is completed. This is because when the header statistics read from memory are added, deleted, or modified, the underlying header information will be modified synchronously during object add, delete, and modify operations, resulting in a mismatch between the header information in memory and the header information written to disk. If the migration is not locked and is directly transferred to the database, the header statistics in the database will not match the actual data.

[0134] Next, the header data of each shard in the target bucket can be migrated. Before migrating the header, the metadata of the target bucket that is not read-based needs to be locked, meaning that add, delete, and modify operations on objects within the target bucket are not allowed during the header migration process. After locking the metadata of the object index pool that is not read-based, the header data of each shard of the target bucket in the object index pool is migrated to the database. Since a bucket typically has 2048 shards, and each shard corresponds to one header statistic, the number is small and the migration can be completed within 1 second. Therefore, sacrificing short-term business operations is necessary to ensure the consistency of the header data.

[0135] See Figure 9 , Figure 9 This diagram illustrates the locking mechanism used during header migration. During header migration, a dual-write, single-read mode is maintained, meaning that metadata can be written to both the database and the object index pool, but only read from the object index pool. Reading metadata from the object index pool is unaffected; however, any add, delete, or modify operations on metadata in the object index pool or the database will fail due to the lock on the target bucket.

[0136] Since header migration is a brief process, a retry mechanism has been added to the add, delete, and modify business processes. Once the header migration is complete and the target bucket is unlocked, the retry succeeds, and there is no actual impact on normal user business. During the header migration process while the bucket is locked, user read requests are not affected in any way.

[0137] For example, during the migration of header data from shards 1 to 9, adding, deleting, and modifying metadata in the object index pool and database is not allowed, but reading data from the object index pool is permitted.

[0138] Furthermore, after migrating the header, the single-write, single-read switch of the target bucket should be switched first. This means that all CRUD operations on the target bucket will affect the metadata in the database, and will no longer affect the data in the object index pool. Only then can the target bucket be unlocked to restore normal CRUD operations. From this point on, all CRUD operations on the target bucket will be handled by the database, and will no longer affect the metadata in the object index pool.

[0139] Therefore, compare_obj can be updated as follows:

[0140]

[0141] In this embodiment of the application, during the second migration phase, the incremental metadata generated by the object index pool during the second migration phase is migrated to the database, and the migration of the header data of each shard in the target bucket is completed.

[0142] Furthermore, in the above embodiments, during the process of synchronizing the metadata of the database based on the forward comparison results and the reverse comparison results between the first metadata and the second metadata stored in the database, that is, in the second migration stage, it is necessary to add a deletion mark to the metadata to be deleted; wherein, the metadata to be deleted is the metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is the metadata shared by the database and the object index pool.

[0143] For example, in the second migration phase, adding deletion tags to the metadata to be deleted in the database is done for the following reasons:

[0144] At the start of the second migration phase, when the bucket operates on the database side, mark-and-delete is enabled. That is, after the start of the second migration phase, all deletion operations for the target bucket are still double deletions, that is, the corresponding metadata in the object index pool and the database are deleted at the same time, but the metadata attributes in the database are marked for deletion, and the metadata is not deleted immediately; the corresponding metadata in the object index pool is actually deleted.

[0145] This is because no locking was used when comparing metadata. If 1000 records are read from the object index pool, and a user issues a delete operation just before comparing the metadata in the database, and the deletion is not marked, two scenarios will occur:

[0146] Scenario 1: In the dual-write-single-read mode, metadata can be written to both the database and the object index pool, but only read from the object index pool. If metadata is successfully deleted from both the object index pool and the database, but 1000 metadata entries read from the object index pool are in memory, the metadata will be migrated from memory to the database. This results in the database ultimately containing dirty data that should have been deleted, which users will be aware of. This dirty data will be cleaned up in the third migration stage, so no invalid dirty data will ultimately be generated. However, during the time span from the beginning of the second migration stage to the end of the third migration stage, there may be instances where dirty data is perceived by users.

[0147] Scenario 2: After the target bucket switches to a read-only, write-only database mode, all data creation, deletion, modification, and query operations apply to the metadata in the database. These operations no longer affect the data in the object index pool. All business operations access the database, so deletion is changed to delete only the metadata in the database. The same metadata in the object index pool will not be deleted. If the database's explicit delete function is not enabled at this stage, the metadata in the database will be deleted, but the metadata in the object index pool will not. In this case, the object index pool contains more metadata than the database, making it impossible to determine whether to migrate to the database in the third migration stage.

[0148] Therefore, if the marking and deletion of metadata in the database is enabled, after switching to a read-only and write-only database, the deletion operation will mark the metadata attribute that is hit in the database as marked for deletion, and will also update the mtime of the metadata. The metadata of the marked object will not be perceived by the user. So after the metadata comparison is completed in the third migration stage, the metadata marked for deletion in the marking and deletion database can be directly cleaned up. The above two problems can be solved by marking and deletion.

[0149] In this embodiment of the application, by adding a deletion mark to the metadata to be deleted in the second migration stage, the problem of unreliable comparison results can be avoided when data in the database is deleted but data in the object index pool is not deleted after the single read and single write database mode is enabled.

[0150] In some optional implementations, this application embodiment uses the third migration stage as an example for illustration. Assuming the third migration stage is the time period after time T2, the first metadata to be compared is all metadata stored in the object index pool up to the end time of the second migration stage, that is, all metadata stored before time T2.

[0151] Based on this, see Figure 10 , Figure 10 Another flowchart for synchronizing metadata in a database is provided, which includes the following steps:

[0152] S1001, compare the first metadata with all metadata stored in the database to obtain a positive comparison result.

[0153] For example, the second migration phase has completed the switch to a read-only, write-only database mode, but the third migration phase is still necessary. In this phase, the second metadata is the full metadata in the database. By comparing the object index pool with the full metadata in the database, the differing metadata is further processed to ensure that the metadata in the database is up-to-date and free of dirty data residue.

[0154] Simultaneously, after changing the marked deletion to actual deletion for the target bucket, the marked deletion metadata in the database is cleaned up. In the second migration phase, the time T2 for switching to the single-read, single-write database mode has been recorded in current_compare_time and persisted. In the third migration phase, the criterion for determining whether inconsistent metadata should be deleted on the database side is whether the metadata's mtime is after T2.

[0155] For example, all the first metadata in the object index pool can be extracted into memory, as well as all the metadata stored in the database can be extracted into memory, and the first metadata can be compared with all the metadata stored in the database to obtain a positive comparison result.

[0156] S1002, if the forward comparison result shows that the fifth target metadata does not exist in the first metadata in the database, synchronize the fifth target metadata to the database.

[0157] If the positive comparison result shows that the fifth target metadata in the first metadata does not exist in the database, and the update time of the fifth target metadata is less than the start time of the third migration phase, which means that the metadata failed to be written to the database in the second migration phase, then the fifth target metadata can be synchronized to the database.

[0158] For example, see Figure 11 , Figure 11 A schematic diagram of the forward comparison in the third migration stage is provided. In this diagram, the mtime of Obj800 and Obj804 in the object index pool is less than T2, and neither was found in the database; therefore, they are considered valid metadata and can be migrated to the database. Obj4 and Obj803 are metadata present in both the database and the object storage pool, and do not require processing. Obj1, Obj3, Obj5, and Obj7 represent operations performed on the database after time T2.

[0159] S1003, compare all metadata stored in the database with the first metadata to obtain the reverse comparison result.

[0160] Furthermore, all metadata stored in the database can be compared with the first metadata to obtain a reverse comparison result.

[0161] S1004, if the reverse comparison result shows that the sixth target metadata exists in the database, but the sixth target metadata does not exist in the first metadata, delete the sixth target metadata.

[0162] For example, if the reverse comparison result shows that the sixth target metadata exists in the database, but the sixth target metadata does not exist in the first metadata, and the update time of the sixth target metadata is less than the start time of the third migration stage, then the sixth target metadata is considered to be dirty data and can be deleted.

[0163] See Figure 12 , Figure 12 A schematic diagram of the reverse comparison in the third migration stage is provided. Among them, Obj741 is dirty data, which can be directly cleared; however, if mtime is after T2, such as Obj43, no processing is performed; if the same metadata exists on both sides, such as Obj34 and Obj502, the data is considered to be consistent and no processing is performed.

[0164] S1005, delete the metadata in the database that is marked for deletion.

[0165] Furthermore, the mark-to-delete flag for the target bucket can be removed, and the marked deletion can be changed to an actual deletion. For example, the mark_del field can be set to false. Subsequent deletion operations on this target bucket will be processed in the database and will actually delete metadata, removing the metadata marked for deletion from the database.

[0166] See Figure 13 , Figure 13 This document provides a schematic diagram for cleaning up metadata marked for deletion. Obj5, Obj133, Obj121, and Obj412 are metadata marked for deletion and are actually deleted in the database; Obj41 and Obj743 represent the actual deletion operations after the target bucket's deletion mark is removed, and Obj41 and Obj743 are actually deleted from the database.

[0167] Furthermore, compare_obj can be initialized to allow data migration from other buckets in the object index pool. This involves initializing compare_obj as follows:

[0168]

[0169] In this embodiment of the application, during the third migration stage, the incremental metadata generated by the object index pool before the third migration stage is migrated to the database, and dirty data and metadata marked for deletion in the database are cleared. This achieves the purpose of migrating the metadata in the object storage pool to the database, and there is no missing metadata or redundant dirty data in the database.

[0170] In some optional implementations, in the above embodiments, if the forward comparison result shows that the first metadata exists in the second metadata, the metadata with the latest update time between the first metadata and the second metadata is synchronized to the database. For example, if the forward comparison result shows that the first metadata exists in the second metadata, the update time mtime of that metadata can be compared between the object index pool and the database. The metadata with the latest update time mtime is used as the standard, and then the metadata with the latest update time mtime is stored in the database.

[0171] In this embodiment of the application, when the same metadata exists in both the object index pool and the database, storing the metadata with the latest update time in the database can ensure that the version of the metadata stored in the database is the latest.

[0172] In some alternative implementations, multiple concurrent threads can be used to perform data comparison tasks in order to improve data comparison efficiency.

[0173] Based on this, see Figure 14 , Figure 14 A flowchart illustrating the process of performing a data comparison task using multi-threading is provided, specifically including the following steps:

[0174] S1401, obtain the number of threads used to compare the first metadata and the second metadata.

[0175] For example, the number of threads used to compare the first and second metadata is the same as the number of threads used for forward comparison. This number can be obtained first.

[0176] S1402, based on the number of shards and the number of threads in the target bucket of the object index pool, determine the shards allocated to each thread for comparison with the second metadata.

[0177] Furthermore, based on the number of shards and the number of threads in the target bucket of the object index pool, the shards allocated to each thread for comparison with the second metadata can be determined. For example, the quotient of the number of shards in the target bucket and the number of threads can be rounded down to ensure that the number of shards allocated to each thread is as balanced as possible.

[0178] For example, see Figure 15 , Figure 15This document provides a diagram illustrating the allocation of task shards to each thread. Assuming 128 shards and 13 threads, an even distribution algorithm can be used to calculate the task list for each thread, storing the shard identity (ID). For example, 10 shards might be allocated to the first 12 threads, and 8 shards to the last thread, aiming for a balanced distribution of shards across threads. Then, thread tasks are created, and each thread concurrently processes the sharded tasks in its respective task list.

[0179] S1403 controls each thread to perform a forward comparison operation between the first metadata and the second metadata in the corresponding slice, and obtains the forward comparison result.

[0180] Furthermore, each thread is controlled to perform a forward comparison operation between the first metadata and the second metadata in its assigned shard, thereby obtaining the forward comparison result. This improves the efficiency of the forward comparison.

[0181] S1404, based on the number of second metadata and the number of threads, determine the second metadata allocated to each thread for comparison with the first metadata.

[0182] For example, the number of threads used to compare the second metadata with the first metadata is the same as the number of threads used for reverse comparison. Similarly, the amount of second metadata allocated to each thread for comparison with the first metadata can be determined based on the amount of second metadata and the number of threads. For instance, the amount of second metadata allocated to each thread can be rounded down by dividing the amount of second metadata by the number of threads to ensure a more balanced distribution across threads.

[0183] For example, see Figure 16 , Figure 16 This document provides a diagram illustrating the allocation of secondary metadata to each thread. Since the database lacks the concept of sharding, it's impossible to partition the subtask list using shards. Therefore, the subtasks are partitioned using [start_key, end_key], and assembled into multiple sets of [start_key, end_key] combinations. These multiple sets are then assigned to their respective thread tasks. Here, start_key and end_key are the index values ​​of the partitioning interval, allowing the corresponding secondary metadata to be retrieved based on these interval index values.

[0184] In this system, the first set of `start_key` is empty, indicating that the query starts from index 0. The last set of `end_key` is empty, indicating that the query ends at the end. Within the thread task, data is enumerated based on the given `start_key` and `end_key`. If the last enumerated key is lexicographically greater than or equal to `end_key`, the thread task can terminate. This ensures that each concurrent thread processes the same second metadata entry as much as possible.

[0185] In this case, assuming there are 10,000 secondary metadata entries and 10 threads, an equal distribution algorithm can be used to allocate 1,000 secondary metadata entries to each thread, so as to make the amount of secondary metadata allocated to each thread as balanced as possible.

[0186] S1405, control each thread to perform a reverse comparison operation between the corresponding second metadata and the first metadata, and obtain the reverse comparison result.

[0187] Furthermore, each thread can be controlled to concurrently perform a reverse comparison operation between the corresponding second metadata and the first metadata to obtain the reverse comparison result. This improves the efficiency of the reverse comparison.

[0188] In this embodiment, by employing multiple concurrent threads to execute forward comparison tasks and reverse comparison tasks, the efficiency of forward and reverse comparison is improved while ensuring that the tasks allocated to each thread are balanced.

[0189] In some alternative implementations, see [link to relevant documentation]. Figure 17 , Figure 17 An alternative data synchronization process diagram is provided, which includes the following steps:

[0190] S1701, obtain the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration phase, and the second metadata corresponding to the database during the target migration phase.

[0191] S1702, obtain the number of threads used to compare the first metadata and the second metadata.

[0192] S1703, Based on the number of shards and the number of threads in the target bucket of the object index pool, determine the shards allocated to each thread for comparison with the second metadata.

[0193] S1704 controls each thread to perform a forward comparison operation between the first metadata and the second metadata in the corresponding slice, and obtains the forward comparison result.

[0194] S1705, based on the number of second metadata and the number of threads, determine the second metadata allocated to each thread for comparison with the first metadata.

[0195] S1706 controls each thread to perform a reverse comparison operation between the corresponding second metadata and the first metadata, and obtains the reverse comparison result.

[0196] S1707, Synchronize the metadata of the database based on the forward comparison results and the reverse comparison results between the first metadata and the second metadata stored in the database.

[0197] This application's embodiment, in the process of uploading large amounts of file data, no longer relies on the concept of sharded metadata storage. From a design perspective, it avoids the business bottleneck of large OMAP alarms and significant decreases in enumeration efficiency when the number of objects in a single bucket increases, and the response efficiency for lifecycle-related business operations is also faster. Furthermore, this application's embodiment adopts the current phased bidirectional multi-concurrency comparison and migration scheme, which not only ensures metadata consistency but also significantly improves metadata comparison and migration efficiency, preventing the potential impact of a lengthy migration process on the original cluster business.

[0198] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0199] Based on the same inventive concept, this application also provides a data synchronization device for implementing the data synchronization method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more data synchronization device embodiments provided below can be found in the limitations of the data synchronization method described above, and will not be repeated here.

[0200] In one embodiment, such as Figure 18 As shown, a data synchronization device is provided, comprising:

[0201] The acquisition module 10 is used to acquire the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein, the target migration stage is the first migration stage, the second migration stage, or the third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage.

[0202] The synchronization module 20 is used to synchronize the metadata of the database based on the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata.

[0203] The aforementioned data synchronization device acquires the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage. The target migration stage can be a first migration stage, a second migration stage, or a third migration stage. The end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage. Based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database, metadata synchronization is performed on the database. The forward comparison result checks whether the first metadata exists within the second metadata; the reverse comparison result checks whether the second metadata exists within the first metadata. This scheme, by comparing the first metadata in the object index pool with the second metadata in the database in stages and synchronizing the first metadata to the database, uses database storage for metadata, avoiding bucket storage, thus solving the problem of too many or too few shards within a bucket. Furthermore, the forward and reverse comparisons ensure that all first metadata in the object index pool is stored in the database, and that there is no dirty data in the database.

[0204] In one embodiment, when the target migration stage is the first migration stage, the first metadata is the metadata stored in the object index pool before the start of the first migration stage; the synchronization module 20 is specifically used for:

[0205] The first metadata is compared with the metadata stored in the database before the start of the first migration phase to obtain a forward comparison result. If the forward comparison result shows that the first target metadata in the first metadata does not exist in the database, the first target metadata is synchronized to the database. All metadata stored in the database is compared with the first metadata to obtain a reverse comparison result. If the reverse comparison result shows that the second target metadata exists in the database and the second target metadata does not exist in the first metadata, the second target metadata is deleted from the database.

[0206] In one embodiment, when the target migration stage is the second migration stage, the first metadata is the metadata stored in the object index pool between the start time and the end time of the first migration stage; the synchronization module 20 is specifically used for:

[0207] The first metadata is compared with the metadata stored in the database between the start time and the end time of the first migration phase to obtain a positive comparison result;

[0208] If the forward comparison result shows that the third target metadata does not exist in the first metadata in the database, the third target metadata is synchronized to the database; the metadata stored in the database from the start time to the end time of the first migration phase is compared with the first metadata to obtain the reverse comparison result; if the reverse comparison result shows that the fourth target metadata exists in the database and the fourth target metadata does not exist in the first metadata, the fourth target metadata is deleted from the database; after locking the metadata that is not read in the target bucket, the header data of each shard in the target bucket is migrated to the database.

[0209] In one embodiment, the device further includes an adding module for:

[0210] During the process of synchronizing the database metadata based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database, in the second migration stage, a deletion mark is added to the metadata to be deleted; wherein, the metadata to be deleted is the metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is the metadata shared by the database and the object index pool.

[0211] In one embodiment, when the target migration stage is the third migration stage, the first metadata is all metadata stored in the object index pool up to the end of the second migration stage; the synchronization module 20 is specifically used for:

[0212] The first metadata is compared with all metadata stored in the database to obtain a forward comparison result. If the forward comparison result shows that the fifth target metadata in the first metadata does not exist in the database, the fifth target metadata is synchronized to the database. The update time of the fifth target metadata is less than the start time of the third migration phase. All metadata stored in the database is compared with the first metadata to obtain a reverse comparison result. If the reverse comparison result shows that the sixth target metadata exists in the database and the sixth target metadata does not exist in the first metadata, the sixth target metadata is deleted. The update time of the sixth target metadata is less than the start time of the third migration phase. The metadata marked for deletion in the database is deleted.

[0213] In one embodiment, the synchronization module 20 is further configured to:

[0214] If the positive comparison result shows that the first metadata exists in the second metadata, then the metadata with the latest update time in the first metadata and the second metadata will be synchronized to the database.

[0215] In one embodiment, the device further includes a comparison module for:

[0216] Obtain the number of threads used for comparing the first and second metadata; determine the shards allocated to each thread for comparing with the second metadata based on the number of shards and threads in the target bucket of the object index pool; control each thread to perform a forward comparison operation between the first and second metadata in the corresponding shard to obtain the forward comparison result; determine the second metadata allocated to each thread for comparing with the first metadata based on the number of second metadata and threads; control each thread to perform a reverse comparison operation between the corresponding second metadata and the first metadata to obtain the reverse comparison result.

[0217] Each module in the aforementioned data synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0218] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 19 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores object data and object metadata. The network interface communicates with external terminals via a network connection. The computer program, when executed by the processor, implements a data synchronization method.

[0219] Those skilled in the art will understand that Figure 19 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0220] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data synchronization method described in any of the above embodiments.

[0221] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the data synchronization method described in any of the above embodiments.

[0222] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the data synchronization method described in any of the above embodiments.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0224] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0225] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0226] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data synchronization method, characterized in that, The method includes: Obtain the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein, the target migration stage is a first migration stage, a second migration stage, or a third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage. Based on the forward comparison results and reverse comparison results between the first metadata and the second metadata stored in the database, the database is synchronized with metadata; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata; When the target migration stage is the third migration stage, the first metadata is all metadata stored in the object index pool up to the end of the second migration stage; the second metadata is all metadata stored in the database. The step of synchronizing the database based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database includes: The first metadata is compared with all metadata stored in the database to obtain a positive comparison result; If the forward comparison result indicates that the fifth target metadata in the first metadata does not exist in the database, the fifth target metadata is synchronized to the database; wherein the update time of the fifth target metadata is less than the start time of the third migration phase; The first metadata is compared with all the metadata stored in the database to obtain the reverse comparison result; If the reverse comparison result indicates that the sixth target metadata exists in the database, but the sixth target metadata does not exist in the first metadata, the sixth target metadata is deleted; wherein the update time of the sixth target metadata is less than the start time of the third migration phase; Delete the metadata in the database that is marked for deletion.

2. The method according to claim 1, characterized in that, When the target migration stage is the second migration stage, the first metadata is the metadata stored in the object index pool between the start time of the first migration stage and the end time of the first migration stage; The second metadata is the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase; The step of synchronizing the database based on the forward and reverse comparison results between the first metadata and the second metadata stored in the database includes: The first metadata is compared with the metadata stored in the database between the start time of the first migration phase and the end time of the first migration phase to obtain a positive comparison result; If the forward comparison result indicates that the third target metadata in the first metadata does not exist in the database, the third target metadata will be synchronized to the database. The metadata stored in the database between the start time and the end time of the first migration phase is compared with the first metadata to obtain a reverse comparison result; If the reverse comparison result indicates that the fourth target metadata exists in the database, but the fourth target metadata does not exist in the first metadata, then the fourth target metadata is deleted from the database. After locking the metadata of the target bucket that is not being read, the header data of each shard of the target bucket is migrated to the database.

3. The method according to claim 2, characterized in that, In the process of synchronizing metadata of the database based on the forward comparison results and the reverse comparison results between the first metadata and the second metadata stored in the database, the method further includes: In the second migration phase, deletion tags are added to the metadata to be deleted; The metadata to be deleted refers to the metadata in the database corresponding to the data deletion operation, and the metadata to be deleted is metadata shared by the database and the object index pool.

4. The method according to claim 1, characterized in that, The method further includes: If the forward comparison result shows that the first metadata exists in the second metadata, the first metadata and the metadata with the latest update time in the second metadata are synchronized to the database.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the number of threads used to compare the first metadata and the second metadata; Based on the number of shards in the target bucket of the object index pool and the number of threads, determine the shards allocated to each thread for comparison with the second metadata. Control each thread to perform a forward comparison operation between the first metadata in the corresponding slice and the second metadata, and obtain the forward comparison result; Based on the quantity of the second metadata and the quantity of threads, determine the second metadata allocated to each thread for comparison with the first metadata; Control each thread to perform a reverse comparison operation between the corresponding second metadata and the first metadata, and obtain the reverse comparison result.

6. The method according to claim 1, characterized in that, When the target migration phase is the first migration phase, the first metadata is the metadata stored in the object index pool before the start time of the first migration phase; the second metadata is the metadata stored in the database before the start time of the first migration phase.

7. A data synchronization device, characterized in that, The device includes: The acquisition module is used to acquire the first metadata corresponding to each shard of the target bucket in the object index pool during the target migration stage; wherein the target migration stage is a first migration stage, a second migration stage, or a third migration stage, the end time of the first migration stage is the start time of the second migration stage, and the end time of the second migration stage is the start time of the third migration stage. The synchronization module is used to synchronize the database metadata based on the forward comparison result and the reverse comparison result between the first metadata and the second metadata stored in the database; wherein, the forward comparison result is the result of comparing whether the first metadata exists in the second metadata; the reverse comparison result is the result of comparing whether the second metadata exists in the first metadata; When the target migration stage is the third migration stage, the first metadata is all metadata stored in the object index pool up to the end of the second migration stage; the second metadata is all metadata stored in the database; the synchronization module is specifically used for: The first metadata is compared with all metadata stored in the database to obtain a forward comparison result. If the forward comparison result indicates that the fifth target metadata in the first metadata does not exist in the database, the fifth target metadata is synchronized to the database. The update time of the fifth target metadata is less than the start time of the third migration phase. All metadata stored in the database is compared with the first metadata to obtain a reverse comparison result. If the reverse comparison result indicates that the sixth target metadata exists in the database and the sixth target metadata does not exist in the first metadata, the sixth target metadata is deleted. The update time of the sixth target metadata is less than the start time of the third migration phase. The metadata marked for deletion in the database is then deleted.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images