A data synchronization method and system for object storage

Abstract

The present invention discloses a data synchronization method and system for object storage, and relates to the field of distributed network storage technology. The method includes: establishing a data synchronization service process for each object storage cluster, configuring the service addresses of the source object storage cluster and the main object storage cluster, and starting the data synchronization service process; according to the synchronization progress of the current object storage cluster, pulling bucket metadata and object metadata lists from the source object storage cluster in batches, and constructing the bucket full mapping log and object full mapping log of the current object storage cluster; according to the mapping log, synchronizing the data of the source object storage cluster to the current object storage cluster in full; when the full synchronization is completed, periodically pulling the local synchronization log from the source object storage cluster and writing it into the local synchronization log of the current object storage cluster; according to the local synchronization log, synchronizing the data increment of the source object storage cluster to the current object storage cluster. The present invention can efficiently store and synchronize massive data.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed network storage, and in particular to a data synchronization method and system for object storage. Background Art

[0002] Object-based Storage (OBS) is a data storage architecture for storing large-scale unstructured data. It designates each piece of data as an object, stores it in a separate container (bucket), and then bundles it with metadata and a unique identifier for easy access and retrieval. With the advent of the big data era, object storage systems have gradually become popular for storing massive and diverse data.

[0003] In the field of distributed network storage, in order to ensure the data security in the distributed network storage system, the stored data may need to be stored in multiple data centers. There are two main modes for data synchronization between data nodes in multiple data centers: active push and asynchronous pull.

[0004] The advantages of the active push mode are: 1. The data nodes that actively push can monitor data changes in real time, and are more efficient when the number of data nodes that passively receive data is small; 2. The implementation method can be quasi-real-time synchronous push or asynchronous periodic push. The disadvantages are: 1. When the number of data nodes that passively receive data is large, the pressure on the data nodes that actively push is too great. 2. When actively pushing data, if the data nodes that passively receive data are busy processing, it will further cause pressure on the data nodes that passively receive data.

[0005] The advantages of asynchronous pull mode are: 1. The data node pulling data can pull data from the master data node according to its busyness and when it is not busy; 2. The master data node can also decide whether to send the data to be synchronized according to its busyness. The disadvantages are: 1. The implementation method is asynchronous periodic pull, and the delay time of data synchronization is limited by the pull cycle.

[0006] In traditional object storage, data is usually written from the master data node, which then actively pushes the data to other data nodes to complete data synchronization. This traditional scenario can meet general needs when there are fewer data nodes to be synchronized. When there are more data nodes to be synchronized, once the amount of data pushed increases, it will lead to a series of problems such as excessive pressure on the master node data synchronization, slower data synchronization, and troublesome subsequent processing of push exceptions. Therefore, an efficient and complete data synchronization method is urgently needed. Summary of the invention

[0007] The purpose of the present invention is to provide a data synchronization method and system for object storage, so as to solve at least one of the above-mentioned coexisting technical problems that the traditional object storage method is difficult to deal with when facing massive data synchronization. The various technical effects that can be produced by the preferred technical solution among the various technical solutions provided by the present invention are described in detail below.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention provides a method for synchronizing object storage data, comprising the following steps:

[0010] Establish a data synchronization service process for each object storage cluster, configure the addresses of the data synchronization services of the source object storage cluster and the master object storage cluster, and start the data synchronization service process;

[0011] According to the synchronization progress of the current object storage cluster, when it is in the initial state, the bucket metadata list and object metadata list are pulled from the source object storage cluster in batches to build the full mapping log of the buckets and the full mapping log of the objects of the current object storage cluster; according to the constructed mapping log, the metadata of the buckets, the metadata and data of the objects in the source object storage cluster are fully synchronized to the current object storage cluster by pulling. During the full synchronization process, the synchronization service process will update the full progress mark in real time. Once the synchronization service process stops abnormally, when it is restarted, it will continue to pull the data after the full progress mark from the source object storage cluster. After the full synchronization is completed, the synchronization progress status is updated to the incremental synchronization status;

[0012] When the full synchronization is completed, the local synchronization log will be periodically pulled from the source object storage cluster and written into the local synchronization log of the current object storage cluster; based on the written local synchronization log, the metadata of the buckets, the metadata of the objects and the data in the source object storage cluster will be incrementally synchronized to the current object storage cluster by pulling. During the incremental synchronization process, the synchronization service process will update the incremental progress mark in real time. Once the synchronization service process stops abnormally, when it is restarted, it will continue to pull the data after the incremental progress mark from the source object storage cluster. The synchronization service process will remain in the incremental synchronization state;

[0013] The local synchronization log of the current object storage cluster can be pulled by the target object storage cluster as the incremental synchronization log of the target object storage cluster. According to the data synchronization method, multiple target object storage clusters can complete full synchronization and incremental synchronization from the current object storage cluster according to the local synchronization log of the current object storage cluster.

[0014] In all embodiments, before starting the data synchronization service process of the current object storage cluster, the following items are configured:

[0015] The N object storage clusters include 1 primary object storage cluster and N-1 backup object storage clusters; the current object storage cluster is the primary object storage cluster or the backup object storage cluster; the TCP service address of the data synchronization service process of the source object storage cluster; the TCP service address of the data synchronization service process of the primary object storage cluster; the number of execution task threads Nt of the data synchronization service process of the current object storage cluster; and the network processing thread Nn of the data synchronization service process of the current object storage cluster.

[0016] When starting the data synchronization service process of the current object storage cluster, the following steps are included:

[0017] Create 1 manager thread, Nt task execution threads, Nn network processing threads, and 1 local synchronization log pruning thread.

[0018] After starting the data synchronization service process of the current object storage cluster, the synchronization progress detection process is performed, including the following steps:

[0019] Execute the synchronization progress detection process, detect the synchronization progress status of the current object storage cluster, and when the synchronization progress is in the initial state, if the current object storage cluster is configured as the backup object storage cluster, the synchronization progress is updated to build a full mapping log of the bucket, and the current object storage cluster pulls the metadata list of the bucket from the source object storage cluster in batches, and executes the construction of the full mapping log of the bucket; if the current object storage cluster is configured as the primary object storage cluster, the synchronization progress is updated to build a full mapping log of the object, and the current object storage cluster pulls the metadata list of the object from the source object storage cluster in batches, and executes the construction of the full mapping log of the object;

[0020] When the synchronization progress is in the full synchronization state of the bucket, the full synchronization process of the bucket in the data synchronization method is executed;

[0021] When the synchronization progress is in the full synchronization state of the object, the full synchronization process of the object in the data synchronization method is executed;

[0022] When the synchronization progress is in the incremental synchronization state, the incremental synchronization process in the data synchronization method is executed.

[0023] When the current object storage cluster constructs the full mapping log of the bucket and the full mapping log of the object, performs the full synchronization of the bucket and the full synchronization of the object, the following steps are included:

[0024] The data synchronization service process of the current object storage cluster initiates a request to the data synchronization service process of the source object storage cluster to obtain a list of TCP service addresses of the storage service processes of the source object storage cluster; after successful acquisition, the TCP service address of the first storage service process is selected from the list, and a TCP network connection is established with it;

[0025] If the storage service process of the source object storage cluster fails and causes the failure to pull the bucket metadata list and the object metadata list, the data synchronization service process of the current object storage cluster selects the TCP service address of the next storage service process from the TCP service address list of the storage service process of the source object storage cluster, establishes a TCP network connection with it, and retries to pull the bucket metadata list and the object metadata list.

[0026] The data synchronization service process includes a management data change process, and executing the management data change process includes the following steps:

[0027] When a user initiates a request to write or modify bucket metadata, object metadata, and data to the storage service process of the current object storage cluster, the storage service process sends a data change request to the data synchronization service process of the current object storage cluster, and the data synchronization service process writes the received data change request to the local synchronization log of the current object storage cluster;

[0028] If the current object storage cluster is configured as a backup object storage cluster, the data synchronization service process of the current object storage cluster writes the bucket metadata change request into the local synchronization log and forwards the request to the primary object storage cluster, which completes writing or modifying the bucket metadata.

[0029] The data synchronization service process includes a local synchronization log management function, which includes a local synchronization log retrieval service, a latest synchronization progress acquisition service, and a local synchronization log clipping service;

[0030] The current object storage cluster provides a local synchronization log retrieval service function, such as Fig. 9 As shown, the target object storage cluster can use this function to pull the local synchronization log of the current object storage cluster after the specified time, and perform incremental synchronization of the target object storage cluster according to the data synchronization method;

[0031] The current object storage cluster provides a service function for obtaining the latest synchronization progress, and other object storage clusters can retrieve the latest data synchronization time of the current object storage cluster;

[0032] The execution of trimming the local synchronization log includes the following steps:

[0033] The current object storage cluster periodically obtains the latest data synchronization time of the target object storage cluster, calculates the earliest data synchronization time of all target object storage clusters, and trims the local synchronization logs before the earliest data synchronization time in the current object cluster.

[0034] In all embodiments, the current object storage cluster can only pull data from one of the source object storage clusters; the current object storage cluster can pull data from at least one of the target object storage clusters; wherein the pulled data includes the bucket metadata list, object metadata list, local synchronization log, bucket metadata, object metadata and data of the object storage cluster.

[0035] According to another aspect of the present invention, there is provided a data synchronization system for object storage, which is applied to the data synchronization method for object storage described above, and includes a source object storage cluster, a current object storage cluster, a target object storage cluster and a synchronization process configuration module; the synchronization process configuration module is used to establish a data synchronization service process for the source object storage cluster, the current object storage cluster and the target object storage cluster, and configure the address of the data synchronization service of the source object storage cluster and the primary object storage cluster; the full synchronization and incremental synchronization of the bucket metadata, object metadata and object data among the source object storage cluster, the current object storage cluster and the target object storage cluster are completed through the data synchronization service process.

[0036] Implementing one of the above technical solutions of the present invention has the following advantages or beneficial effects:

[0037] The present invention can be used for data synchronization between multiple object storage clusters, and solves at least the following seven technical problems:

[0038] 1. It can complete data synchronization between 2 to N object storage clusters and effectively share the data synchronization pressure of a single object storage cluster;

[0039] 2. By establishing a full mapping of buckets and a full mapping log of objects, the problem of changes in the metadata of buckets or metadata and data of objects in the source object storage cluster during full data synchronization is solved;

[0040] 3. Support breakpoint continuation, which solves the problem of unexpected failure of the data synchronization service process. After the data synchronization service process is restarted, it will continue to pull from the interrupted position (full progress mark / incremental progress mark);

[0041] 4. Support batch pulling, which solves the problem of insufficient memory resources and busy CPU resources caused by pulling too much data during data synchronization;

[0042] 5. Highly available data synchronization: when a storage service process of the source object storage cluster fails, the data synchronization service process of the current object storage cluster will select the TCP service of the next storage service process from the TCP service address list of the storage service process of the source object storage cluster;

[0043] 6. Solved the data loop problem during two-way data synchronization;

[0044] 7. Support hot synchronization. During the full synchronization and incremental synchronization of the current object storage cluster, the current object storage cluster and the source object storage cluster can still provide storage services to users normally.

[0045] Therefore, the present invention designs a method for synchronizing data between multiple object storage clusters, and can effectively share the pressure brought by the synchronization of massive data in the object storage cluster; the present invention designs and implements a system for synchronizing data between multiple object storage clusters, solves the technical problems, and is effectively applied to multiple object storage clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following will introduce the drawings required for describing the embodiment, in which:

[0047] Figure 1 Schematic diagram of data synchronization applicable scenarios of an embodiment of the present invention (taking three object storage clusters as an example);

[0048] Figure 2 This is a flow chart of a data synchronization method for object storage according to an embodiment of the present invention;

[0049] Figure 3 This is a flow chart of a manager thread detecting synchronization progress of a data synchronization service process according to an embodiment of the present invention;

[0050] Figure 4 This is a flowchart of building a full mapping log of a bucket according to an embodiment of the present invention;

[0051] Figure 5 is a flowchart of constructing a full mapping log of an object according to an embodiment of the present invention;

[0052] Figure 6 is a flowchart of performing full synchronization of a bucket according to an embodiment of the present invention;

[0053] Figure 7 is a flowchart of a full synchronization of an execution object according to an embodiment of the present invention;

[0054] Figure 8It is a flow chart of creating a synchronization object in a full synchronization program of an object in an embodiment of the present invention;

[0055] Fig. 9 It is a flow chart of determining and replacing objects in a full synchronization program of objects in an embodiment of the present invention;

[0056] Fig.10 It is a flow chart of determining and updating object metadata in a full synchronization program of an object in an embodiment of the present invention;

[0057] Fig.11 is a flow chart of performing incremental synchronization according to an embodiment of the present invention;

[0058] Fig.12 is a schematic diagram of a data synchronization system structure of an object storage according to an embodiment of the present invention;

[0059] Fig.13 It is a schematic diagram of an object storage cluster-data synchronization service process managing data changes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, devices, etc. that are consistent with some aspects of the present disclosure as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present invention.

[0061] In the description of the present invention, it should be understood that the term "plurality" means two or more than two. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0062] In order to illustrate the technical solution of the present invention, a specific embodiment is used below for description, and only the parts related to the embodiment of the present invention are shown.

[0063] Implementation 1:

[0064] like Figure 1 As shown, the present invention provides a data synchronization method for object storage, comprising the following steps:

[0065] S100: Establish a data synchronization service process for each object storage cluster, configure the addresses of the data synchronization services of the source object storage cluster and the master object storage cluster, and start the data synchronization service process.

[0066] S200. According to the synchronization progress of the current object storage cluster, when it is in the initial state, the bucket metadata list and object metadata list are pulled from the source object storage cluster in batches to build the full mapping log of the buckets and the full mapping log of the objects of the current object storage cluster; according to the constructed mapping log, the metadata of the buckets, the metadata and data of the objects in the source object storage cluster are fully synchronized to the current object storage cluster by pulling. During the full synchronization process, the synchronization service process will update the full progress mark (full mapping, full synchronization progress mark) in real time. Once the synchronization service process stops abnormally, when it is restarted, it will continue to pull the data after the full progress mark from the source object storage cluster. After the full synchronization is completed, the synchronization progress status is updated to the incremental synchronization status.

[0067] S300. When the full synchronization is completed, the local synchronization log will be periodically pulled from the source object storage cluster and written into the local synchronization log of the current object storage cluster; according to the written local synchronization log, the metadata of the buckets, the metadata of the objects and the data in the source object storage cluster will be incrementally synchronized to the current object storage cluster by pulling. During the incremental synchronization process, the synchronization service process will update the incremental progress mark (the progress mark of incremental synchronization) in real time. Once the synchronization service process stops abnormally, when it is restarted, it will continue to pull the data after the incremental progress mark from the source object storage cluster. The synchronization service process will remain in the incremental synchronization state.

[0068] The local synchronization log of the current object storage cluster can be pulled by the target object storage cluster as the incremental synchronization log of the target object storage cluster. According to the data synchronization method, multiple target object storage clusters can complete full synchronization and incremental synchronization from the current object storage cluster based on the local synchronization log.

[0069] The data synchronization method of this embodiment is applicable to data synchronization between N object storage clusters. Taking three object storage clusters as an example, its applicable scenarios are as follows: Figure 2 As shown, there are 1 primary object storage cluster and 2 backup object storage clusters. In the figure, the 3 object storage clusters are: object storage cluster A, object storage cluster B and object storage cluster C, where object storage cluster A is the primary object storage cluster (abbreviated as primary in the figure), object storage cluster B is a backup object storage cluster (abbreviated as backup 1 in the figure), and object storage cluster C is another backup object storage cluster (abbreviated as backup 2 in the figure). Typical application scenarios of data synchronization include:

[0070] One master and multiple backups - master-slave unidirectional synchronization: the data of the master object storage cluster is pulled by the two backup object storage clusters;

[0071] One master and multiple backups - chained one-way synchronization: the data of the master object storage cluster A is pulled by the backup object storage cluster B (backup 1), and the data of the backup object storage cluster B (backup 1) is pulled by the backup object storage cluster C (backup 2);

[0072] One master and multiple backups - chained bidirectional synchronization: The data of the master object storage cluster A is pulled by the backup object storage cluster B (backup 1), and the data of the backup object storage cluster B (backup 1) is pulled by the master object storage cluster A and the backup object storage cluster C (backup 2).

[0073] It should be noted that, in a specific embodiment, the N object storage clusters include 1 primary object storage cluster and N-1 backup object storage clusters. Among them, any object storage cluster (current object storage cluster) can only pull data from one source object storage cluster; the current object storage cluster can be pulled by multiple target object storage clusters.

[0074] It should be noted that there are mainly two types of data that need to be synchronized in the object storage cluster: one is the bucket metadata corresponding to the bucket (storage bucket), which is the container of the object and records metadata such as access control permissions, policies, and quotas; the other is the object data and object metadata corresponding to the object, which is the basic unit of data storage and contains object data and object metadata. The data in the synchronization, storage, modification, creation, and other processes of this embodiment can all be referred to as these two types of data.

[0075] The data synchronization service process (sync-server) includes the synchronization progress detection process, full synchronization process, incremental synchronization process, data change management process, and local synchronization log management functions.

[0076] Before starting the data synchronization service process of the current object storage cluster, configure the following items:

[0077] The current object storage cluster is a primary object storage cluster or a backup object storage cluster;

[0078] The TCP service address of the data synchronization service process of the source object storage cluster;

[0079] The TCP service address of the data synchronization service process of the primary object storage cluster;

[0080] The number of execution task threads Nt of the data synchronization service process of the current object storage cluster;

[0081] The number of network processing threads Nn of the data synchronization service process of the current object storage cluster.

[0082] When starting the data synchronization service process of the current object storage cluster (hereinafter referred to as the data synchronization service process sync-server), the following steps are included:

[0083] Create 1 manager thread, Nt task execution threads, Nn network processing threads, and 1 local synchronization log pruning thread.

[0084] After the sync-server is started, the main tasks of the manager thread include: periodically correcting the system time error value between the current object storage cluster and the source object storage cluster; and periodically detecting the data synchronization progress status.

[0085] Periodically correct the system time difference between the current object storage cluster and the source object storage cluster, including:

[0086] The manager thread of the sync-server of the current object storage cluster initiates a request to the sync-server of the source object storage cluster every period Ct, obtains the current system time Tsource of the source object storage cluster, and caches it in the process memory; the sync-server of the current object storage cluster updates the cache every Ct, and compares it with the current system time Tlocal, calculates the time difference between Tsource and Tlocal, which is the corrected system time error value Tdiff. In a specific embodiment, the default period value of Ct is 1 hour, which can be customized.

[0087] It should be noted that the sync-server of each object storage cluster provides a service interface for obtaining the current system time. Other object storage clusters can obtain the current system time of the cluster where the sync-server is located through this interface.

[0088] Periodically check the data synchronization progress status, including:

[0089] The manager thread of the sync-server of the current object storage cluster executes a synchronization progress detection every period Cchk. In a specific embodiment, the default period value of Cchk is 2 seconds, which can be customized.

[0090] like Figure 3 As shown in the figure, the process of the sync-server manager thread detecting the synchronization progress includes the following steps:

[0091] S110: Determine the state of the manager thread.

[0092] S120. If the state of the manager thread is the stopped state, the manager thread dispatches Nt thread exit tasks to the Nt task execution threads in sequence. When the Nt task execution threads have completed the execution of existing tasks and have exited respectively, the manager thread finally cleans up the thread resources and exits.

[0093] S130: If the manager thread is in the running state, the synchronization progress is checked every period Cchk, and the synchronization progress information is retrieved from the metadata repository of the current object storage cluster.

[0094] S140: Determine whether synchronization progress information exists.

[0095] S150: If the synchronization progress information does not exist, create the synchronization progress information. Creating the synchronization progress information includes the following steps:

[0096] S151. The manager thread initiates a request to the sync-server of the source object storage cluster to obtain the latest timestamp Trlog of the source object storage cluster log.

[0097] S152, judging whether the latest timestamp Trlog is obtained successfully.

[0098] S153. If the acquisition fails, set the thread state to the stopped state.

[0099] S154. Otherwise, if the current object storage cluster is the primary object storage cluster, set the synchronization progress status to build the full mapping log of the object; if the current object storage cluster is the backup object storage cluster, set the synchronization progress status to build the full mapping log of the bucket.

[0100] S155: insert the latest timestamp Trlog of the source object storage cluster log and the synchronization progress status of the current object storage cluster into the metadata database of the current object storage cluster as synchronization progress information.

[0101] S160: If the synchronization progress information exists, the process of dispatching the data synchronization task is carried out according to the synchronization progress status. The process of dispatching the data synchronization task includes the following steps:

[0102] The manager thread determines the current synchronization progress state, and after completing the corresponding data synchronization processing task, it advances to the next synchronization progress state in sequence, and finally remains in the incremental synchronization state. The synchronization progress states include: building a full mapping log of a bucket, building a full mapping log of an object, executing a full synchronization of a bucket, executing a full synchronization of an object, and executing incremental synchronization. The first four states belong to the full synchronization stage.

[0103] It is understandable that the sync-server of the current object storage cluster supports users to write data when performing full synchronization. Therefore, full synchronization is to synchronize the buckets and object data of the source object storage cluster to the current object storage cluster. It will not delete the existing data of the current object storage cluster, but will update the old data that already exists in the current object storage cluster.

[0104] The full synchronization phase is divided into four sub-steps: building a full mapping log for buckets, building a full mapping log for objects, executing full synchronization of buckets, and executing full synchronization of objects. It should be noted that if the sync-server of the current object storage cluster belongs to the backup object storage cluster, it will enter the process of building a full mapping log for buckets; if the sync-server of the current object storage cluster belongs to the primary object storage cluster, it will enter the process of building a full mapping log for objects.

[0105] like Figure 4 As shown in the figure, building a full mapping log for a bucket includes the following steps:

[0106] S201. The manager thread of the sync-server of the current object storage cluster initiates a request to the sync-server of the source object storage cluster to obtain the TCP service address list of the storage service process (storage service process, hereinafter referred to as storage-server) of the source object storage cluster. After successful acquisition, the TCP service of the first storage service process is selected from the list and a TCP network connection is established with it.

[0107] S202. If there is a full progress mark, initiate a request to the storage-server of the source object storage cluster to pull the bucket list after the full progress mark in batches; otherwise, pull the bucket list in batches from the beginning.

[0108] It should be noted that the pulled bucket list is a collection, each element of which contains the bucket name and bucket metadata. The number of bucket lists pulled in each batch is 1,000 by default and can be customized. Pulling in batches solves the problem of insufficient memory resources and busy CPU resources caused by pulling too much data during data synchronization.

[0109] S203. If the bucket list of the batch is successfully pulled, a bucket mapping log is generated for each element in the bucket list in sequence and inserted into the metadata database of the current object storage cluster. The generation method of each bucket mapping log is as follows:

[0110] Calculate the md5 hash value of the bucket metadata;

[0111] Bucket mapping log format is: bucket name + md5 (bucket metadata)

[0112] According to the above format, each bucket mapping log is generated in the order of elements in the bucket list.

[0113] It should be noted that if a storage-server of the source object storage cluster fails and causes a failure in pulling the bucket list, the sync-server of the current object storage cluster will select the TCP service of the next storage-server from the TCP service address list of the storage service process of the source object storage cluster, establish a TCP network connection with it, and return to step S202.

[0114] S204: Update the full progress mark to the last bucket in the batch bucket list.

[0115] S205: Determine whether the full mapping log of the bucket has been constructed. The determination is based on whether the bucket list of the batch is empty.

[0116] S206: If the full mapping log of the bucket has not been built, continue to pull the bucket list of the next batch after the full progress mark, and return to step S202 until the full mapping log of the bucket is built.

[0117] S207: If the full mapping log of the bucket is successfully constructed, the synchronization progress status is updated to the full mapping log of the constructed object, and the full progress mark is cleared.

[0118] like Figure 5 As shown in the figure, building the full mapping log of the object includes the following steps:

[0119] S208, the manager thread of the sync-server of the current object storage cluster initiates a request to the sync-server of the source object storage cluster to obtain the TCP service address list of the storage-server of the source object storage cluster. After successful acquisition, the TCP service of the first storage service process is selected from the list and a TCP network connection is established with it.

[0120] S209: traverse the full mapping log of the bucket, select each bucket in turn, and build a full mapping log of the object for each bucket.

[0121] S210. If there is a full progress mark, initiate a request to the storage-server of the source object storage cluster to pull the object list of the selected bucket after the full progress mark in batches; otherwise, pull the object list of the selected bucket in batches from the beginning.

[0122] It should be noted that the pulled object list is a collection, each element of which contains the object name, the hash value of the object data, and the object metadata. The number of object lists pulled in each batch is 1,000 by default and can be customized. Pulling in batches solves the problem of insufficient memory resources and busy CPU resources caused by pulling too much data during data synchronization.

[0123] S211. If the batch object list of the selected bucket is successfully pulled, an object mapping log is generated for each element in the object list in sequence and inserted into the metadata database of the current object storage cluster. The generation method of each object mapping log is as follows:

[0124] Calculate the md5 hash value of the object metadata;

[0125] The object mapping log format is: bucket name + object name + hash value of object data + md5 (object metadata);

[0126] According to the above format, each object mapping log is generated in the order of elements in the object list.

[0127] It should be noted that if a storage-server of the source object storage cluster fails and causes a failure in pulling the object list, the sync-server of the current object storage cluster will select the TCP service of the next storage-server from the TCP service address list of the source object storage cluster storage service process, establish a TCP network connection with it, and return to step S210.

[0128] S212: Update the full progress mark to the last object in the batch object list of the selected bucket.

[0129] It should be noted that if the full object mapping log of the selected bucket has not been built, the object list of the next batch of the selected bucket will continue to be pulled from after the full progress mark, and the process returns to step S210 until the full object mapping log of the selected bucket is built.

[0130] S213: Determine whether the full object mapping logs of all buckets have been constructed. The determination is based on whether the batch object list of the selected bucket is empty.

[0131] S214: If the full object mapping logs of all buckets have not been constructed, the next bucket of the full mapping log of the bucket is used as the selected bucket, and the process returns to step S209 until the full object mapping logs of all buckets are constructed.

[0132] S215. If the full object mapping logs of all buckets are successfully constructed and the current object storage cluster is the primary object storage cluster, the synchronization progress status is updated to the full synchronization object and the full progress mark is cleared; otherwise, the synchronization progress status is updated to the full synchronization bucket and the full progress mark is cleared.

[0133] like Figure 6 As shown in the figure, executing full synchronization of buckets includes the following steps:

[0134] S216. The manager thread of the sync-server of the current object storage cluster initiates a request to the sync-server of the primary object storage cluster to obtain a TCP service address list of the storage-server of the primary object storage cluster.

[0135] S217, the manager thread of sync-server traverses the full mapping log of the bucket, selects each bucket in turn, uses the mapping log information of each bucket as the synchronization task, selects one thread from Nt execution task threads in a polling manner, and dispatches the synchronization task to it.

[0136] S218 , the execution task thread of the sync-server receives the synchronization task of the bucket from the manager thread, and obtains the mapping log of the selected bucket Bs.

[0137] S219: Initiate a request to the storage-server of the source object storage cluster to retrieve metadata of the selected bucket Bs.

[0138] It should be noted that if a storage-server of the source object storage cluster fails and fails to pull the metadata of the selected bucket, the sync-server of the current object storage cluster will select the TCP service of the next storage-server from the TCP service address list of the storage service process of the source object storage cluster, establish a TCP network connection with it, and retry to pull the metadata of the selected bucket.

[0139] S220, if the result of pulling is that the selected bucket Bs in the source object storage cluster no longer exists, then notify the manager thread that the bucket in the source object storage cluster no longer exists; and suspend the current task execution thread to wait for receiving the next task. Otherwise, proceed to the next step (step S221).

[0140] S221. If the current object storage cluster has a bucket Bl with the same name, update the bucket metadata, step S221-2; otherwise, create the bucket metadata, step S221-1.

[0141] S221-1. Create bucket metadata, including the following steps:

[0142] According to the metadata of the selected bucket Bs pulled, the metadata of the bucket Bl with the same name is created in the current object storage cluster; the local synchronization log of the bucket Bl with the same name is written in the current object storage cluster, and the content of the local synchronization log is: current system timestamp + operation type (create bucket) + bucket Bl name + bucket Bl metadata md5 hash value; notify the manager thread: the synchronization task is executed successfully; and suspend the current task execution thread, waiting to receive the next task.

[0143] S221-2, updating bucket metadata, including the following steps:

[0144] 1) Calculate the metadata md5 hash value of the bucket Bl with the same name. If the metadata hash value of the bucket Bl with the same name is the same as that of the selected bucket Bs, notify the manager thread that the metadata of the bucket Bl with the same name is the latest; and suspend the current task execution thread to wait for the next task. Otherwise, initiate a request to the storage-server of the primary object storage cluster to pull the metadata of the bucket Bm with the same name as the bucket Bl with the same name.

[0145] It should be noted that if a storage-server of the primary object storage cluster fails and fails to pull the metadata of the bucket with the same name, the sync-server of the current storage cluster will select the TCP service of the next storage-server from the TCP service address list of the storage service process of the primary object storage cluster, establish a TCP network connection with it, and retry to pull the metadata of the bucket with the same name.

[0146] 2) Based on the metadata of the bucket Bm pulled, update the metadata of Bl in the current object storage cluster; write the local synchronization log of the bucket Bl with the same name in the current object storage cluster. The content of the local synchronization log is: current system timestamp + operation type (bucket modification) + name of the bucket Bl with the same name + md5 hash value of the metadata of the bucket Bl with the same name; notify the manager thread: the synchronization task is executed successfully; and suspend the current task execution thread, waiting to receive the next task.

[0147] When the manager thread of the sync-server receives the result notification of the task execution thread, it performs the following processing: if the notification result is that the bucket of the source object storage cluster no longer exists, the metadata of the bucket Bl with the same name is the latest, or the synchronization task is successfully executed, the mapping log of the selected bucket Bs is deleted. If there are still mapping logs of other buckets, the manager thread of the sync-server will return to step S217; otherwise, the synchronization progress status is updated to the full synchronization object.

[0148] like Figure 7-10 As shown, the full synchronization of the execution object includes the following steps:

[0149] S222. The manager thread of the sync-server of the current object storage cluster traverses the full mapping log of the object, selects each object in turn, uses the mapping log information of each object as a synchronization task, selects one thread from Nt task execution threads in a round-robin manner, and dispatches the synchronization task to it.

[0150] S223. The execution task thread of the sync-server receives the object synchronization task from the manager thread, and obtains the mapping log of the selected object Os.

[0151] S224: Initiate a request to the storage-server of the source object storage cluster to retrieve metadata of the selected object Os.

[0152] It should be noted that if a storage-server of the source object storage cluster fails and the metadata of the selected object fails to be pulled, the sync-server of the current object storage cluster will select the TCP service of the next storage-server from the TCP service address list of the storage service process of the source object storage cluster, establish a TCP network connection with it, and retry to pull the metadata of the selected object.

[0153] S225, if the result of pulling is that the object Os in the source object storage cluster no longer exists, the manager thread is notified that the object in the source object storage cluster no longer exists, and the current task execution thread is suspended to wait for receiving the next task. Otherwise, the next step (step S226) is entered.

[0154] S226. If the current object storage cluster has an object with the same name as Ol, then determine and replace the object, step S226-2, or determine and update the object metadata, step S226-3; otherwise, create a synchronization object, step S226-1.

[0155] S226-1. Creating a synchronization object includes the following steps:

[0156] Initiate a request to the storage-server of the source object storage cluster to pull the data and metadata of the selected object Os.

[0157] If the data and metadata of the selected object Os are successfully pulled, the data and metadata of the object Ol with the same name are created in the current object storage cluster; the local synchronization log of the object Ol with the same name is written in the current object storage cluster, and the content of the local synchronization log is: current system timestamp + operation type (create object) + name of the bucket to which object Ol belongs + name of object Ol + md5 hash value of metadata of object Ol + md5 hash value of data of object Ol; notify the manager thread: synchronization task execution is successful; and suspend the current task execution thread and wait for the next task to be received. If the data and metadata of object Os are pulled, the current task execution thread is suspended and wait for the next task to be received.

[0158] S226-2, determining and replacing the object, including the following steps:

[0159] If the data hash value of the object Ol with the same name is the same as that of the selected object Os, indicating that the data of the object Ol with the same name is consistent with that of the selected object Os, then step S226-3 is executed; otherwise, the corrected value Tcos of the creation time of the selected object Os is calculated, and the calculation method is:

[0160] Tcos = creation time of Os in the source object storage cluster ± system time error value Tdiff;

[0161] If the creation time of the object Ol with the same name is newer than Tcos, it means that the data of the object Ol with the same name is the latest, and then step S226-3 is executed; otherwise, the process proceeds to the next step.

[0162] Delete the data and metadata of the object Ol with the same name in the current object storage cluster and execute step S226-1.

[0163] S226-3, determining and updating object metadata, including the following steps:

[0164] If the metadata hash value of the object Ol with the same name is the same as that of the selected object Os, indicating that the metadata of the object Ol with the same name is consistent with that of the selected object Os, the manager thread is notified that the object Ol with the same name is the latest one, and the currently executing task thread is suspended to wait for the next task to be received; otherwise, the corrected value Tuos of the update time of the selected object Os is calculated, and the calculation method is:

[0165] Tuos = update time of Os in the source object storage cluster ± system time error value Tdiff;

[0166] If the update time of the object Ol with the same name is newer than Tuos, it means that the metadata of the object Ol with the same name is the latest, then the manager thread is notified: the object Ol with the same name is the latest; and the current task execution thread is suspended to wait for receiving the next task;

[0167] Otherwise, based on the metadata of the selected object Os, update the metadata of the object Ol with the same name in the current object storage cluster; write the local synchronization log of the object Ol with the same name in the current object storage cluster, and the content of the local synchronization log is: current system timestamp + operation type (updated object) + name of the bucket to which object Ol belongs + object Ol name + md5 hash value of metadata of object Ol; notify the manager thread: the synchronization task is executed successfully; and suspend the current task execution thread, waiting to receive the next task.

[0168] When the manager thread of the sync-server receives the result notification of the task execution thread, it performs the following processing: if the notification result is that the object in the source object storage cluster no longer exists, the object Ol with the same name is the latest, or the synchronization task is successfully executed, then the mapping log of the selected object Os is deleted. If there are still mapping logs of other objects, the manager thread of the sync-server returns to step S222; otherwise, the synchronization progress status is updated to incremental synchronization.

[0169] It should be noted that through the three processes S226-1, S226-2, and S226-3, the hash values ​​of the object data and metadata are verified to compare whether the objects with the same name from the two object storage clusters are consistent; if they are inconsistent, they will be replaced or updated with the latest object. This processing method solves the data loop problem in the two-way data synchronization process.

[0170] It is understandable that when the sync-server of the current object storage cluster performs incremental synchronization, it synchronizes the metadata, object data, and metadata increments of the buckets in the source object storage cluster to the current object storage cluster, and also updates or deletes the old data of the current object storage cluster. The incremental synchronization process is also applicable to multiple target object storage clusters performing incremental synchronization from the current object cluster.

[0171] like Fig.11 As shown, performing incremental synchronization includes the following steps:

[0172] S227. The manager thread of the sync-server sets the latest timestamp Trlog of the source object storage cluster log as the incremental progress mark.

[0173] S228, the manager thread of the sync-server initiates a request to the sync-server of the source object storage cluster to pull the synchronization logs after the incremental progress mark in batches. Otherwise, each synchronization log that has been pulled is selected in turn, and it is used as a synchronization task. A thread is selected from the Nt execution task threads in a polling manner, and the synchronization task is dispatched to it.

[0174] It should be noted that the default number of synchronization logs pulled in each batch is 100, which can be customized. If the synchronization log pull fails, the manager thread of sync-server will continue to pull the synchronization log after the incremental progress mark when the synchronization progress is checked next time.

[0175] S229. The execution task thread of the sync-server receives the incremental synchronization task from the manager thread, and obtains the incremental synchronization log of the selected object Os.

[0176] S230 . Execute a corresponding process according to the operation type of the incremental synchronization log of the selected object Os.

[0177] Specifically, the operation types and corresponding processes include:

[0178] S230-1. If the operation type is to create a bucket, perform the following steps:

[0179] If there is no bucket Bl with the same name in the current object storage cluster, execute step S221-1 of creating bucket metadata; otherwise, notify the manager thread that Bl is the latest, and suspend the current task execution thread to wait for receiving the next task.

[0180] S230-2: If the operation type is to modify the bucket, execute step S221-2 of updating the bucket metadata.

[0181] S230-3: If the operation type is to delete a bucket, the following steps are included:

[0182] If there is no bucket Bl with the same name in the current object storage cluster, notify the manager thread that Bl is the latest; and suspend the current task execution thread to wait for receiving the next task; otherwise, proceed to the next step.

[0183] If the object exists in Bl, notify the manager thread: the operation failed; and suspend the current task execution thread to wait for receiving the next task; otherwise, go to the next step.

[0184] Notify the manager thread that the synchronization task has been successfully executed; and suspend the current task execution thread to wait for receiving the next task.

[0185] S230-4. If the operation type is to create an object, the following steps are included:

[0186] If the current object storage cluster has an object with the same name Ol, then the step S226-2 of determining and replacing the object is executed; otherwise, the step S226-1 of creating a synchronization object is executed.

[0187] S230-5: If the operation type is to update the object, execute step S226-3 of determining and updating the object metadata.

[0188] S230-6. If the operation type is to delete an object, the following steps are included:

[0189] Calculate the corrected value Tdos of the Os deletion time using the following calculation method:

[0190] Tdos=deletion time of Os in the source object storage cluster±system time error value Tdiff.

[0191] If the update time of Ol is newer than Tdos, the manager thread is notified that Ol is the latest one and the current task execution thread is suspended to wait for the next task.

[0192] Otherwise, delete the data and metadata of Ol in the current object storage cluster. Notify the manager thread that the synchronization task is successfully executed; and suspend the current task execution thread to wait for the next task.

[0193] When the manager thread of the sync-server receives the result notification of the task execution thread, it performs the following processing: if the notification result is that the object in the source object storage cluster no longer exists, Ol is already the latest, or the synchronization task is successfully executed, the count value Ninc is accumulated.

[0194] If the number of incremental logs pulled for this batch is equal to Ninc, it means that the incremental synchronization of this batch is successful, and the manager thread of the sync-server sets the latest time of the incremental logs of this batch as the incremental progress mark; otherwise, the manager thread of the sync-server will dispatch the unsuccessful incremental synchronization task again.

[0195] The manager thread of sync-server will enter the next synchronization progress detection and perform the next batch of incremental synchronization.

[0196] It is understandable that when the sync-server of the target object storage cluster performs incremental synchronization, the steps of performing incremental synchronization by the sync-server of the above-mentioned object storage cluster may be referred to, and will not be repeated here.

[0197] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0198] Embodiment 2:

[0199] like Figure 12-13 As shown, this embodiment provides an object storage data synchronization system, which is applied to an object storage data synchronization method described in the embodiment, including a source object storage cluster, a current object storage cluster, a target object storage cluster and a synchronization process configuration module. Among them, the source object storage cluster, the current object storage cluster, and the target object storage cluster are all composed of multiple storage devices for storing object data, and each storage device stores bucket metadata, object metadata and object data.

[0200] The synchronization process configuration module is used to establish a data synchronization service process for each object storage cluster and configure the service addresses of the source object storage cluster and the primary object storage cluster; through the data synchronization service process, the full and incremental synchronization of bucket metadata, object metadata and object data between the source object storage cluster, the current object storage cluster and the target object storage cluster is completed.

[0201] It is understandable that the applicable scenario of this embodiment is consistent with that of Embodiment 1. The above-mentioned object storage cluster should be generally understood to include but not limited to products such as servers and storage devices.

[0202] This embodiment takes object storage cluster A as an example to illustrate the process of object storage cluster management object data changes. Object storage cluster A obtains objects uploaded by users through the object storage service process. When the object storage service process writes to the storage device of the current cluster, it will notify its data synchronization service process, and the data synchronization service process will write to the local synchronization log, thereby forming a data synchronization log used by this data synchronization system to perform as described in Example 1.

[0203] In summary, this system is used to synchronize data between multiple object storage clusters and can effectively share the pressure brought by the synchronization of massive cluster data; through communication and interaction with the source object storage cluster, the data of the source object storage cluster can be synchronized to the current object storage cluster; by providing a local synchronization log function, the target object storage cluster can synchronize the data of the current object storage cluster to the target object storage cluster.

[0204] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. A data synchronization method for object storage, characterized in that: The steps include: Establish a data synchronization service process for each object storage cluster, configure the addresses of the data synchronization services of the source object storage cluster and the master object storage cluster, and start the data synchronization service process; According to the synchronization progress of the current object storage cluster, when it is in the initial state, the bucket metadata list and object metadata list are pulled from the source object storage cluster in batches to build the full mapping log of the bucket and the full mapping log of the object of the current object storage cluster; according to the constructed mapping log, the metadata of the bucket, the metadata and data of the object in the source object storage cluster are fully synchronized to the current object storage cluster by pulling; When the full synchronization is completed, the local synchronization log is periodically pulled from the source object storage cluster and written into the local synchronization log of the current object storage cluster; according to the written local synchronization log, the metadata of the buckets, the metadata and the data of the objects in the source object storage cluster are incrementally synchronized to the current object storage cluster by pulling; The local synchronization log of the current object storage cluster can be pulled by the target object storage cluster as the incremental synchronization log of the target object storage cluster.

2. The method for synchronizing object storage data according to claim 1, characterized in that: Before starting the data synchronization service process of the current object storage cluster, the following items are configured: The current object storage cluster is a primary object storage cluster or a backup object storage cluster; The TCP service address of the data synchronization service process of the source object storage cluster; The TCP service address of the data synchronization service process of the primary object storage cluster; The N object storage clusters include 1 primary object storage cluster and N-1 backup object storage clusters.

3. The method for synchronizing object storage data according to claim 2, characterized in that: The data synchronization service process includes a synchronization progress detection process, and after starting the data synchronization service process of the current object storage cluster, the process includes the following steps: Execute the synchronization progress detection process to detect the synchronization progress status of the current object storage cluster. When the synchronization progress is in the initial state, if the current object storage cluster is configured as the standby object storage cluster, the synchronization progress is updated to build a full mapping log of the bucket. The current object storage cluster pulls the metadata list of the bucket from the source object storage cluster in batches and executes the construction of the full mapping log of the bucket. If the current object storage cluster is configured as the primary object storage cluster, the synchronization progress is updated to build a full mapping log of the object, and the current object storage cluster pulls the metadata list of the object from the source object storage cluster in batches to execute the construction of the full mapping log of the object.

4. The method for synchronizing object storage data according to claim 2, characterized in that: When constructing the full mapping log of the bucket and the full mapping log of the object of the current object storage cluster, the following steps are included: The data synchronization service process of the current object storage cluster initiates a request to the data synchronization service process of the source object storage cluster to obtain a TCP service address list of the storage service process of the source object storage cluster; After the acquisition is successful, select the TCP service address of the first storage service process from the list and establish a TCP network connection with it; If the storage service process of the source object storage cluster fails and causes the failure to pull the bucket metadata list and the object metadata list, the data synchronization service process of the current object storage cluster selects the TCP service address of the next storage service process from the TCP service address list of the storage service process of the source object storage cluster, establishes a TCP network connection with it, and retries to pull the bucket metadata list and the object metadata list.

5. The method for synchronizing object storage data according to claim 2, characterized in that: The data synchronization service process includes a management data change process, and executing the management data change process includes the following steps: When a user initiates a request to write or modify bucket metadata, object metadata, and data to the storage service process of the current object storage cluster, the storage service process sends a data change request to the data synchronization service process of the current object storage cluster, and the data synchronization service process writes the received data change request to the local synchronization log of the current object storage cluster; If the current object storage cluster is configured as a backup object storage cluster, the data synchronization service process of the current object storage cluster writes the bucket metadata change request into the local synchronization log and forwards the request to the primary object storage cluster, which completes writing or modifying the bucket metadata.

6. The method for synchronizing object storage data according to claim 1, characterized in that: The data synchronization service process updates the progress marks of full mapping, full synchronization, and incremental synchronization in real time. If the data synchronization service process stops abnormally, after it is restarted, the data synchronization service process of the current object storage cluster continues to pull data after the progress marks of full mapping, full synchronization, and incremental synchronization from the source object storage cluster.

7. The object storage data synchronization method according to claim 1, characterized in that: When starting the data synchronization service process of the current object storage cluster, the following steps are included: Create 1 manager thread, Nt task execution threads, Nn network processing threads, and 1 local synchronization log pruning thread.

8. The method for synchronizing object storage data according to claim 1, characterized in that: The data synchronization service process includes a local synchronization log management function, which includes a local synchronization log retrieval service, a latest synchronization progress acquisition service, and a local synchronization log clipping service; The execution of trimming the local synchronization log includes the following steps: The current object storage cluster periodically obtains the latest data synchronization time of the target object storage cluster, calculates the earliest data synchronization time of all target object storage clusters, and trims the local synchronization logs before the earliest data synchronization time.

9. A method for synchronizing object storage data according to any one of claims 1 to 8, characterized in that: The current object storage cluster can only pull data from one of the source object storage clusters; the current object storage cluster can be pulled data by at least one of the target object storage clusters; The pulled data includes the bucket metadata list of the object storage cluster, the object metadata list, the local synchronization log, the bucket metadata, the object metadata and data.

10. A data synchronization system for object storage, characterized in that: A data synchronization method for object storage as described in any one of claims 1 to 9, comprising a source object storage cluster, a current object storage cluster, a target object storage cluster and a synchronization process configuration module; The synchronization process configuration module is used to establish a data synchronization service process for the source object storage cluster, the current object storage cluster, and the target object storage cluster, and configure the address of the data synchronization service of the source object storage cluster and the primary object storage cluster; The data synchronization service process completes the full synchronization and incremental synchronization of the bucket metadata, object metadata, and object data among the source object storage cluster, the current object storage cluster, and the target object storage cluster.

Images