Data synchronization control method and device, equipment and medium

Abstract

The invention relates to the technical field of distributed storage, and mainly relates to a data synchronization control method and device, equipment and a medium, and the method comprises the steps: determining whether to execute a data synchronization operation or not according to a first log file; under the condition that it is determined that the data synchronization operation is executed, calling a synchronization thread, and obtaining synchronization pipeline information of the bucket to be synchronized; determining whether the bucket synchronization flag bits of the target bucket and the source bucket indicate synchronization pause or not according to the identifier of the target bucket and the identifier of the source bucket; under the condition that the synchronization flag bit of any one of the target bucket and the source bucket indicates synchronization pause, the synchronization process is controlled to skip to execute synchronization operation on the bucket to be synchronized; under the condition that the bucket synchronization flag bits of the target bucket and the source bucket do not indicate synchronization pause, the synchronization process is controlled to execute synchronization operation on the bucket to be synchronized; and updating the first log file. Through the embodiment of the invention, bucket-level data synchronization control is realized, the system can be controlled in a fine-grained manner, and resource allocation of the system is flexibly adjusted.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and mainly to a data synchronization control method, device, equipment and medium. Background Art

[0002] With the explosive growth of massive unstructured data, distributed object storage has become the cornerstone for users to build data centers, and more and more key businesses are connected to distributed object storage. In this scenario, distributed object storage products need to ensure that customers have storage needs for massive data while also meeting disaster recovery and backup needs for key data. For example, in the financial industry, there are multiple data centers in a city, and how to synchronize, backup, and share data among multiple data centers; when a cluster fails, it is necessary to ensure that user businesses continue to run.

[0003] A distributed system with active-active data center features is proposed in the related technology. The active-active data center feature means that two data centers are in operation at the same time and can undertake production business at the same time, thereby greatly improving the overall service capacity and resource utilization. This feature is achieved through two-way synchronization and backup between the two data centers, ensuring business continuity and high availability. Since the multi-site active-active feature will consume a lot of system resources of the entire cluster when it is enabled, when the customer's front-end business pressure is high, it will affect the interaction with the client. Therefore, for the distributed object storage system with multi-site active-active features, how to flexibly control it to optimize the configuration of system resources is an urgent problem to be solved. Summary of the invention

[0004] One of the purposes of the present application is to propose a data synchronization control method to solve or at least partially solve the problems in the related art. The second purpose of the present application is to provide a data synchronization control device, the third purpose of the present application is to provide a device, and the fourth purpose of the present application is to provide a medium.

[0005] In order to achieve the above purpose, the technical solution of this application is as follows: A data synchronization control method, the method comprising: Determining whether to perform a data synchronization operation according to a first log file of the first site; In the case of determining to perform a data synchronization operation, calling a synchronization thread to obtain synchronization pipeline information of a bucket to be synchronized in the first site, wherein the synchronization pipeline information includes identifiers of a target bucket and a source bucket; Determine, according to the identifier of the target bucket and the identifier of the source bucket, whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization pause; When the synchronization flag bit of any bucket in the target bucket and the source bucket indicates synchronization suspension, the synchronization process is controlled to skip the synchronization operation on the bucket to be synchronized; When the bucket synchronization flags of the target bucket and the source bucket do not indicate synchronization pause, controlling the synchronization process to perform a synchronization operation on the bucket to be synchronized; The first log file is updated.

[0006] Optionally, determining whether to perform a data synchronization operation according to the first log file of the first site includes: comparing the first log file with a second log file of a second site; When the value of the update mark of the second log file is higher than the value of the update mark of the first log file, it is determined that the first site needs to perform a data synchronization operation.

[0007] Optionally, in the case of determining to perform a data synchronization operation, calling a synchronization thread to obtain synchronization pipeline information of a bucket to be synchronized in the first site includes: In the case of determining to perform a data synchronization operation, parsing the first log file to determine an identifier of a bucket to be synchronized; According to the identifier of the bucket to be synchronized, the synchronization pipeline information of the bucket to be synchronized is obtained.

[0008] Optionally, determining, according to the identifier of the target bucket and the identifier of the source bucket, whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization suspension includes: According to the identifier of the target bucket and the identifier of the source bucket, acquiring the attribute information of the target bucket and the attribute information of the source bucket; the attribute information of the bucket includes a bucket synchronization flag; According to the value of the bucket synchronization flag bit, it is determined whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization pause.

[0009] Optionally, the method further includes: In the case where the bucket synchronization flag of the bucket to be synchronized is adjusted from indicating synchronization pause to not indicating synchronization pause, the first site sends a recovery request to the second site; the second site includes a source bucket corresponding to the bucket to be synchronized; the recovery request is used to update the value of the update identifier of the second log file of the second site, so that the value of the update identifier of the second log file is higher than the value of the update identifier of the first log file; Determine, according to the updated second log file, a mapping relationship between the second log and a third log in the second site; Determining, according to the mapping relationship, the incremental data of the second site stored in the third log file; Determine the target data in the incremental data according to the synchronization channel information of the bucket to be synchronized; the target data is: the operation data executed by the source bucket corresponding to the bucket to be synchronized during the most recent suspension of the synchronization operation of the bucket to be synchronized; The synchronization thread is called to synchronize the target data into the bucket to be synchronized.

[0010] Optionally, before determining whether to perform a data synchronization operation according to the first log file of the first site, the method further includes: Detecting whether a cluster synchronization flag of the cluster in the first site indicates synchronization suspension; When the cluster synchronization flag indicates synchronization suspension, the synchronization thread corresponding to the cluster whose cluster synchronization flag indicates synchronization suspension is disabled; Stop updating the first log file.

[0011] Optionally, the method further includes: When the cluster synchronization flag of the cluster is adjusted to indicate that the synchronization is not suspended, triggering the first site to perform an update operation on the cluster configuration information; According to the updated cluster configuration information, the cluster synchronization flag is enabled to indicate the synchronization thread corresponding to the unsynchronized paused cluster.

[0012] A data synchronization control device, the device comprising: A first determination module, used to determine whether to perform a data synchronization operation according to a first log file of a first site; An acquisition module, configured to, when determining to perform a data synchronization operation, call a synchronization thread to acquire synchronization pipeline information of a bucket to be synchronized in the first site, wherein the synchronization pipeline information includes identifiers of a target bucket and a source bucket; A second determination module is used to determine whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization suspension according to the identifier of the target bucket and the identifier of the source bucket; A first control module is used to control the synchronization process to skip the synchronization operation on the bucket to be synchronized when the synchronization flag bit of any bucket in the target bucket and the source bucket indicates synchronization suspension; A second control module is used to control the synchronization process to perform a synchronization operation on the bucket to be synchronized when the bucket synchronization flag bits of the target bucket and the source bucket do not indicate synchronization pause; An updating module is used to update the first log file.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data synchronization control method when executing the computer program.

[0014] A non-volatile storage medium stores a computer program, which implements the above-mentioned data synchronization control method when executed by a processor.

[0015] Beneficial effects of this application: In the present application, by determining whether to perform a data synchronization operation according to the first log file of the first site; in the case of determining to perform a data synchronization operation, calling a synchronization thread to obtain the synchronization pipeline information of the bucket to be synchronized in the first site, the synchronization pipeline information includes the identifiers of the target bucket and the source bucket; according to the identifier of the target bucket and the identifier of the source bucket, determining whether the barrel synchronization flag of the target bucket and the source bucket indicates synchronization pause; in the case that the barrel synchronization flag of any one of the target bucket and the source bucket indicates synchronization pause, controlling the synchronization process to skip the synchronization operation on the bucket to be synchronized; in the case that the barrel synchronization flag of the target bucket and the source bucket does not indicate synchronization pause, controlling the synchronization process to perform the synchronization operation on the bucket to be synchronized; updating the first log file. Through the barrel synchronization flag of each barrel, data synchronization control of a single barrel can be realized, and after skipping the synchronization operation on the barrel with synchronization pause, the synchronization operation of other barrels is not affected, and the first log file is still updated, so that the update status between sites remains consistent, so that data synchronization control at the barrel level is realized, the system can be controlled in a fine-grained manner, and the resource allocation of the system can be flexibly adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a flow chart of a data synchronization control method described in an embodiment of the present application; Figure 2 This is a partial logic diagram of the bucket-level synchronization suspension described in an embodiment of the present application; Figure 3 This is a partial logic diagram of bucket-level synchronous recovery described in an embodiment of the present application; Figure 4 This is a partial logic diagram of the global level synchronization suspension described in an embodiment of the present application; Figure 5 This is a partial logic diagram of global level synchronous recovery described in an embodiment of the present application; Figure 6 It is a schematic diagram of the system structure of the multi-site active-active feature described in an embodiment of the present application; Figure 7 Schematic diagram of the framework of the data synchronization control device described in one embodiment of the present application.

[0018] Among them, 700-data synchronization control device, 701-first determination module, 702-acquisition module, 703-second determination module, 704-first control module, 705-second control module, 706-update module. DETAILED DESCRIPTION

[0019] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0020] Distributed object storage products need to meet customers' storage needs for massive data and disaster recovery and backup needs for key data. The common disaster recovery modes in the current market can be divided into scenarios such as disaster recovery in the same city, disaster recovery in different locations, and active-active data centers. Disaster recovery in different locations means that the distance between the primary and backup centers is far (>200KM), so asynchronous mirror replication is generally used. Disaster backup in different locations can not only prevent potential risks such as fire and building damage, but also prevent risks such as war, earthquake, and floods. Since disaster backup in the same city and disaster backup in different locations have their own strengths, in order to achieve the most ideal disaster prevention effect, the data center should consider using the method of establishing a disaster backup center in the same city and in different locations.

[0021] Local disaster recovery refers to the establishment of a disaster recovery system in the local computer room, which can share the operation of the business and management systems at the same time under normal circumstances, and can switch operations; in the event of a disaster, emergency switching of disaster recovery can be performed to maintain continuous business operation. The disaster recovery of the local computer room is mainly used to prevent failures in the production environment, and the off-site disaster recovery center is used to prevent large-scale regional disasters. Since the disaster recovery of the local computer room is in the same computer room as the production center and can be connected through the local area network, data replication and application switching are relatively easy to implement, and real-time replication of data and rapid switching of applications between production and disaster recovery servers can be achieved. In addition, in some scenarios, customers only need to back up part of the data in some buckets, or require the destination bucket to be different from the source bucket, and different data synchronization strategies need to be set based on the bucket.

[0022] For a distributed object storage system with multi-site active-active features, a flexible control synchronization solution is needed, which allows customers to adjust the priority of front-end business and synchronization according to their own business conditions. When the front-end business is prioritized, the synchronization overhead can be suspended, and when the synchronization is enabled again, the data during the suspension period can be pulled back. Based on this, this application proposes a data synchronization control method.

[0023] In the nouns used in this application: zone: a logical object storage cluster, a zone contains multiple RGW (‌Rados Gateway, object storage gateway) instances; zonegroup: contains one or more zones.

[0024] Period: Each period contains a unique ID and epoch. Each commit operation increments the epoch. Each realm is associated with a current period, which maintains the current configuration state of the zonegroup and storage policy; the realm is a large container covering the survival timeline of the entire cluster, while the period is a series of objects used to track configuration changes. When the zone configuration changes, the period may use a new branch to store relevant information.

[0025] Sync policy: synchronization policy.

[0026] pipe: a synchronous pipeline, which contains information such as synchronous flow direction and control conditions. pipes means multiple pipes.

[0027] Bucket: refers to a small storage unit or container in the system; source bucket: refers to the bucket in the pipe that is used to pull data during data synchronization; target bucket: refers to the bucket that is used to pull data during data synchronization.

[0028] Specifically, refer to Figure 1 , Figure 1 1 is a flow chart of a data synchronization control method in an embodiment of the present application. The data synchronization method of the present application is applied to a multi-site active-active distributed storage system, wherein the system includes at least a first site and a second site. Figure 1 As shown, in one embodiment of the present application, the data synchronization control method is specifically performed according to the following steps: Step S101: determining whether to perform a data synchronization operation according to a first log file of a first site.

[0029] Specifically, for the sake of convenience, in a multi-site scenario, it is assumed that the system only includes the first site and the second site, and there is only one zone in each of the first site and the second site, and multiple buckets are stored in the zone. During the synchronization process between the first site and the second site, the first site deploys a first log file. Specifically, the first log file is a datalog file. In a multi-site active-active distributed storage system, the datalog file records the implementation details of the data logic, which can provide a powerful and flexible way to handle complex rules and queries, and also support incremental updates to the database. The datalog file allows developers to process and manipulate data by defining rules and queries, which is essential for managing and maintaining data consistency and integrity in a distributed storage system. Through the datalog file, the flow and processing logic of data can be controlled more finely to ensure the accuracy and reliability of data in a multi-site environment. According to the first log file, it is determined whether a new synchronization operation needs to be performed, that is, according to the datalog file between the first site and the second site, it is determined whether to perform a data synchronization operation.

[0030] In a feasible implementation manner, step S101, determining whether to perform a data synchronization operation according to a first log file of a first site, includes: The first log file is compared with the second log file of the second site; when the value of the update mark of the second log file is higher than the value of the update mark of the first log file, it is determined that the first site needs to perform a data synchronization operation.

[0031] Specifically, the second log file of the second site is the datalog file of the second site. In the actual application of the system, whether to perform the synchronization operation is determined by comparing the values ​​of the respective update marks datalog markers between the sites. If the datalog marker of the first site is higher than the datalog marker of the second site, it indicates that the data version of the first site is updated at this time, and the second site needs to pull the new data of the first site for synchronization. Therefore, when determining whether to perform the synchronization operation, the first log file is compared with the second log file of the second site; determine which of the update mark recorded in the first log file and the update mark recorded in the second log file of the second site is higher. In the case that the value of the update mark of the second log file is higher than the value of the update mark of the first log file, the first site needs to pull the data of the second site for synchronization, and it can be determined that the first site needs to perform a data synchronization operation.

[0032] In this implementation, whether to perform a data synchronization operation and whether to call a synchronization thread for synchronization are determined based on the datalog between system sites, thereby ensuring data consistency between system sites.

[0033] Step S102: When it is determined to perform a data synchronization operation, a synchronization thread is called to obtain synchronization pipeline information of the bucket to be synchronized in the first site.

[0034] The synchronization pipeline information includes identifiers of a target bucket and a source bucket.

[0035] Specifically, when it is determined to perform a data synchronization operation, the synchronization thread is called to first determine the bucket to be synchronized in the first site. For example, the datalog is parsed to determine which buckets have changed. The target bucket corresponding to the bucket with these changed data is the bucket to be synchronized (the bucket to be synchronized needs to pull data from the source bucket of another site for synchronization). After determining the bucket to be synchronized, the synchronization pipeline information configured for the bucket to be synchronized can be obtained. The synchronization pipeline information can indicate the data flow direction, including the identifiers of the target bucket and the source bucket in each synchronization pipeline (such as name, number, etc.).

[0036] Step S103: Determine, according to the identifier of the target bucket and the identifier of the source bucket, whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization pause.

[0037] Specifically, after parsing the synchronization channel information to obtain the identifier of the target bucket and the identifier of the source bucket, determine whether the bucket synchronization flag of each of the target bucket and the source bucket indicates synchronization pause. In the present application, the attributes of each bucket are pre-configured with a bucket synchronization flag. By configuring the bucket synchronization flag, it is controlled whether the bucket performs synchronization pause or normal synchronization. For example, if the bucket synchronization flag of a bucket is configured as 0, it indicates that the bucket performs the synchronization operation normally. If the bucket synchronization flag of a bucket is configured as 1, it indicates that the bucket suspends the synchronization operation. As long as the bucket synchronization flag of any bucket in the source bucket and the target bucket in the synchronization channel indicates synchronization pause, the synchronization channel is disabled, and all operations that originally use the synchronization pipeline for data synchronization are skipped. Therefore, the following step S104 is executed. If the bucket synchronization flags of the source bucket and the target bucket in the synchronization pipeline do not indicate synchronization pause, the following step S105 is executed.

[0038] Step S104: When the synchronization flag of any bucket in the target bucket and the source bucket indicates synchronization suspension, the synchronization process is controlled to skip the synchronization operation on the bucket to be synchronized.

[0039] Specifically, if the synchronization flag of any bucket in the target bucket and the source bucket indicates synchronization suspension, it can be regarded as that the data connection at one end of the synchronization pipeline is disabled. At this time, the synchronization thread no longer uses the synchronization pipeline to synchronize data and skips performing synchronization operations on the bucket to be synchronized.

[0040] Step S105: When the bucket synchronization flags of the target bucket and the source bucket do not indicate synchronization pause, the synchronization process is controlled to perform a synchronization operation on the bucket to be synchronized.

[0041] Specifically, if the bucket synchronization flags of the target bucket and the source bucket do not indicate synchronization pause, the synchronization pipeline can be considered normal. At this time, the synchronization thread can use the synchronization pipeline to synchronize data and perform synchronization operations on the bucket to be synchronized.

[0042] Step S106: updating the first log file.

[0043] Specifically, after performing the above operations on each bucket to be synchronized, the datalog of the first site is updated to keep the update status of the first site consistent with that of other sites. In this way, synchronization suspension control is performed on the buckets configured for synchronization suspension. At this time, these buckets no longer consume system resources and do not affect the normal synchronization operation of other buckets, and system resources can be provided preferentially for the services selected by users.

[0044] In this embodiment, according to the first log file of the first site, it is determined whether to perform the data synchronization operation; in the case of determining to perform the data synchronization operation, the synchronization thread is called to obtain the synchronization pipeline information of the bucket to be synchronized in the first site, and the synchronization pipeline information includes the identifiers of the target bucket and the source bucket; according to the identifier of the target bucket and the identifier of the source bucket, it is determined whether the barrel synchronization flag of the target bucket and the source bucket indicates synchronization suspension; in the case that the barrel synchronization flag of any one of the target bucket and the source bucket indicates synchronization suspension, the synchronization process is controlled to skip the synchronization operation on the barrel to be synchronized; in the case that the barrel synchronization flag of the target bucket and the source bucket does not indicate synchronization suspension, the synchronization process is controlled to perform the synchronization operation on the barrel to be synchronized; and the first log file is updated. Through the barrel synchronization flag of each barrel, data synchronization control of a single barrel can be realized, and after skipping the synchronization operation on the barrel with synchronization suspension, the synchronization operation of other barrels is not affected, and the first log file is still updated, so that the update status between the sites remains consistent, so that the data synchronization control at the barrel level is realized, the system can be controlled in a fine-grained manner, and the resource allocation of the system can be flexibly adjusted.

[0045] It can be understood that in this embodiment, the synchronization control process when the first site pulls the data of the second site for synchronization is described. When the second site pulls the data of the first site for synchronization, the identities of the first site and the second site are exchanged; the first site and the second site can also pull data from each other.

[0046] In a feasible implementation, in the above step S102, when it is determined to perform a data synchronization operation, calling a synchronization thread to obtain synchronization pipeline information of the bucket to be synchronized in the first site includes: In the case of determining to perform a data synchronization operation, parsing the first log file to determine an identifier of a bucket to be synchronized; According to the identifier of the bucket to be synchronized, the synchronization pipeline information of the bucket to be synchronized is obtained.

[0047] Specifically, after determining that the first site needs to perform data synchronization operations according to the datalog, the synchronization thread is called to parse the datalog to determine which buckets have data changes, and then the identifier of the bucket to be synchronized can be determined. According to the identifier of the bucket to be synchronized, the synchronization pipe information is obtained from the configuration information of the bucket to be synchronized.

[0048] In this embodiment, when it is determined to perform a data synchronization operation, the synchronization thread can be called to obtain the synchronization pipeline information of the bucket to be synchronized. In this way, it is convenient to determine whether the buckets at both ends are allowed to perform synchronization operations when pulling data based on the synchronization pipeline information, which is conducive to realizing bucket-level synchronization control.

[0049] In a feasible implementation, determining whether the bucket synchronization flags of the target bucket and the source bucket indicate synchronization pause according to the identifier of the target bucket and the identifier of the source bucket includes: According to the identifier of the target bucket and the identifier of the source bucket, the attribute information of the target bucket and the attribute information of the source bucket are obtained; the attribute information of the bucket includes a bucket synchronization flag; according to the value of the bucket synchronization flag, it is determined whether the bucket synchronization flags of the target bucket and the source bucket respectively indicate synchronization pause.

[0050] Specifically, the bucket attributes of each bucket include a bucket synchronization flag. After obtaining the attribute information of the bucket according to the bucket identifier, the value configured by the bucket synchronization flag can be viewed from the attribute information. If the value corresponds to synchronization pause, the bucket synchronization flag indicates synchronization pause.

[0051] In this embodiment, since the bucket attributes of each bucket include a bucket synchronization flag, whether the bucket needs to be synchronized can be determined based on the bucket identifier. In this way, bucket-level synchronization control can be implemented in the system according to user needs, which facilitates the rational allocation of system resources.

[0052] In a feasible implementation manner, the control method described in the above embodiment further includes: In the case where the bucket synchronization flag of the bucket to be synchronized is adjusted from indicating synchronization pause to not indicating synchronization pause, the first site sends a recovery request to the second site; the second site includes a source bucket corresponding to the bucket to be synchronized; the recovery request is used to update the value of the update identifier of the second log file of the second site, so that the value of the update identifier of the second log file is higher than the value of the update identifier of the first log file; Determine, according to the updated second log file, a mapping relationship between the second log and a third log in the second site; Determining, according to the mapping relationship, the incremental data of the second site stored in the third log file; Determine the target data in the incremental data according to the synchronization pipeline information of the bucket to be synchronized; the target data is: the operation data executed by the source bucket corresponding to the bucket to be synchronized during the most recent suspension of the synchronization operation of the bucket to be synchronized; The synchronization thread is called to synchronize the target data into the bucket to be synchronized.

[0053] Specifically, the bucket synchronization flag of the bucket can be adjusted. If the bucket is allowed to perform the synchronization operation, the bucket synchronization flag of the bucket is configured to not indicate synchronization pause. If it is not allowed, the bucket synchronization flag of the bucket is configured to indicate synchronization pause. Therefore, when the bucket synchronization flag of the bucket to be synchronized is adjusted from indicating synchronization pause to not indicating synchronization pause, it is necessary to start the synchronization operation of the synchronization bucket. In the early stage, in order to keep other normally synchronized buckets unaffected and keep the update status consistent between sites, in order to start the synchronization operation of the bucket, a recovery request is sent to the second site. The second site includes the source bucket corresponding to the bucket to be synchronized, that is, the site where the bucket that the bucket to be synchronized needs to pull data is located, and a recovery request is sent to the site.

[0054] The recovery request is used to update the value of the update marker of the second log file of the second site, so that the value of the update marker of the second log file is higher than the value of the update marker of the first log file. It can be understood that after the second site receives the recovery request, it updates the datalog of the second site. After the update, the datalog marker of the second site is higher than the datalog marker of the first site. At this time, the first site can determine that a data synchronization operation needs to be performed and enable the synchronization thread to perform data synchronization.

[0055] The third log file is a kind of bilog. In actual applications, there is a corresponding index relationship between datalog and bilog, and bilog stores detailed information of each update operation of datalog (such as the object targeted by the update operation, the content of the object, etc.). Therefore, the detailed information of the data change in the second site will be recorded in the bilog. After the synchronization of the synchronization bucket is restored, the incremental data of each bucket of the second site is read from the bilog. At this time, according to the synchronization pipeline information of the bucket to be synchronized, the incremental data of the source bucket corresponding to the bucket to be synchronized in the incremental data is determined (the source bucket corresponding to the bucket to be synchronized, the operation data performed during the most recent suspension of the synchronization operation of the bucket to be synchronized from the current time), that is, the target data. Finally, the synchronization thread is called to synchronize the target data to the bucket to be synchronized.

[0056] In this embodiment, when a bucket is adjusted from synchronization pause to synchronization resumption, the update of datalog is triggered by the resumption request, and then the synchronization thread of the bucket can be started to determine the target data to be synchronized by the bucket for data synchronization. In this way, through this embodiment, not only the synchronization pause control at the bucket level can be realized, but also the synchronization resumption control at the bucket level can be realized accordingly.

[0057] In one embodiment of the present application, based on the bucket-level synchronization control solution, a global-level / cluster-level synchronization control solution is also provided. In a feasible implementation, before determining whether to perform a data synchronization operation according to the first log file of the first site, the method further includes: Detecting whether a cluster synchronization flag of the cluster in the first site indicates synchronization suspension; When the cluster synchronization flag indicates synchronization suspension, the synchronization thread corresponding to the cluster whose cluster synchronization flag indicates synchronization suspension is configured to be disabled; Stop updating the first log file.

[0058] Specifically, in this embodiment, the system cluster is configured with a cluster synchronization flag, and whether to perform synchronization pause control on the cluster can be determined based on the cluster synchronization flag. It can be understood that the cluster synchronization flag of the cluster in the first site is detected to see whether it indicates synchronization pause; if the cluster synchronization flag indicates synchronization pause, the synchronization thread corresponding to the cluster whose cluster synchronization flag indicates synchronization pause is disabled; at this time, the data in the cluster is all suspended from synchronization, the first log file is stopped from being updated, and the data of the site is in a suspended synchronization state.

[0059] In this implementation, the cluster synchronization flag is set to perform synchronization pause control on the cluster, thereby achieving global level bucket synchronization control.

[0060] Following the above implementation, in a feasible implementation, the above method further includes: When the cluster synchronization flag of the cluster is adjusted from indicating synchronization pause to not indicating synchronization pause, configuring the synchronization thread corresponding to the cluster to be in an enabled state; The synchronization thread corresponding to the cluster is called to perform synchronization operations.

[0061] Specifically, when the cluster synchronization flag of the cluster is adjusted to indicate unsynchronized suspension, the update flag is low because the first log file is not updated during the cluster synchronization suspension process.

[0062] In this embodiment, whether to enable the synchronization thread is determined according to the cluster synchronization flag, thereby realizing global-level data synchronization control. When synchronization is suspended at the global level, the synchronization thread is disabled, and no bucket-level synchronization thread is derived to perform data synchronization. When synchronization is resumed at the global level, the synchronization process can continue to identify whether each bucket is suspended synchronously, thereby realizing bucket-level synchronization control.

[0063] In this embodiment, a control scheme for pausing and resuming data synchronization is provided. This scheme supports suspension and resumption control at the bucket granularity level, and also supports synchronization suspension and resumption at the cluster level. After synchronization is paused, this site or this bucket will no longer pull data from other sites or buckets with mapping relationships, and other sites will no longer request data from this site. After synchronization is resumed, the objects uploaded during the pause process can be synchronized according to the synchronization rules. Users are supported to select the current allocation of system resources according to their own business priorities. For example, if the front-end business is prioritized, you can choose to suspend the synchronization business of the entire cluster, or the synchronization business of certain buckets, so as to reasonably allocate system resources and improve system availability.

[0064] For example, in the zone policy, configure the synchronization status of the current cluster. Forbiden means pause, and enabled means pause and resume. However, when set to Forbiden, when the cluster updates the zonegroup related information, it can exclude the cluster from synchronization zones according to this status, and disable the synchronization thread of this cluster, so that the cluster-level synchronization of this site can be suspended. The datalog between sites will not be updated. After re-enabling the enabled status, the synchronization thread of this cluster can still be enabled according to the change of the datalog marker between sites to pull data, so as to synchronize the uploaded objects during the suspension period.

[0065] In order to further illustrate the technical solution of the present application, Figures 2 to 5 Let me give you an example.

[0066] Reference Figure 2 , Figure 2 FIG. 1 is a partial logic diagram of the synchronous suspension of the bucket level described in an embodiment of the present application. Figure 2As shown, a bucket attribute in a site is configured to turn on the multi-site pause switch of the bucket (at this time, the bucket synchronization flag of the bucket indicates synchronization pause); site A needs to pull site B, and determines whether synchronization is required by calculating the size of the update marker; if necessary, parse the names of the source bucket and the target bucket according to the first log file, obtain the synchronization pipeline information, and determine the value of the bucket synchronization flag; determine whether the bucket synchronization flags of the source bucket and the target bucket both indicate unsynchronized pause; if the bucket synchronization flags of the source bucket and the target bucket both indicate unsynchronized pause, perform incremental synchronization of the bucket shards, pull data, and update the update marker of the first log file (update datalog marker); if the bucket synchronization flags of the source bucket and the target bucket do not both indicate unsynchronized pause (any one indicates that synchronization pause is turned on), skip the subsequent data pulling process of the synchronization pipeline, do not perform the synchronization operation, and directly update the update marker of the first log file (update datalog marker).

[0067] Reference Figure 3 , Figure 3 FIG. 1 is a partial logic diagram of the bucket-level synchronous recovery described in an embodiment of the present application. Figure 3 As shown in the figure, if the bucket synchronization flag of a bucket is adjusted from indicating synchronization pause to not indicating synchronization pause (at this time, the synchronization of the bucket is resumed); then the multi-site pipes rule of the bucket (including the multiple synchronization pipeline information of the bucket) is parsed, all source buckets (buckets from which data is pulled) in the synchronization pipeline are traversed, and a recovery request is sent to the zone (region) where each source bucket is located. The datalog of the site where the zone (region) is located is rewritten, and the marker is updated. At this time, if the datalog marker of the site where the zone (region) is located is higher, the site where the bucket is located that is adjusted to synchronization recovery starts the data synchronization operation, and the unsynchronized data of the bucket can be synchronized.

[0068] Exemplarily, in one embodiment of the present application, the main steps of pausing and resuming bucket-level synchronization include: Step 1: Add a bucket synchronization flag to the bucket attributes and do not decode the field in the synchronization decoding to simplify the synchronization logic and reduce possible sources of errors. The undecoded fields can be processed as needed in the subsequent stages to improve the flexibility of the system.

[0069] step2. Add commands to control the settings of the new fields added in step1.

[0070] Step 3: When the synchronization thread executes the scanning datalog shard processing, it finds that the synchronization flag in its own bucket and the target bucket attribute in the synchronization strategy indicates that the synchronization is suspended, so it skips the processing of the shard, updates the datalog marker, and the new data in the bilog stops overwriting the old data.

[0071] If the system includes site A and site B: a. In the scenario of A->B (-> direction is the direction of data pulling), A sets synchronization pause: B traverses the buckets in modify_shards when pulling, and then determines whether the synchronization flag on the bucket indicates a pause. If there is a pause, bilog synchronization is skipped (bilog can add data but no longer perform data overwriting) and the datalog marker is updated (these are the actions performed by site B, the data puller, at this time); b. In the scenario of B->A, A sets synchronization pause: First determine whether A has set a pause. After pulling datalog, A uniformly updates the markers of the shards in modify_shards and also skips the synchronization of bilog (these are the actions performed by the data pulling party A site at this time); c. In the scenario of A->B and B->A, A is set to be paused synchronously: both A and B sites must perform the above actions (sites that pull data from each other must perform corresponding pause actions).

[0072] Step 4, Processing recovery steps: After setting up synchronization recovery on the bucket, the bucket on the corresponding zone needs to record datalog for all shards again (the site where the source bucket is located performs this action) to trigger the bucket synchronization thread of the destination site to pull data. At this time, the bucket that has recovered synchronization can be used as the target bucket to pull the data of the source bucket.

[0073] Reference Figure 4 , Figure 4 This is a partial logic diagram of the global level synchronization suspension described in an embodiment of the present application. Figure 4 As shown, the zones in the system are configured with zone synchronization flags (assuming that the system is configured as the first site and the second site, for a multi-site distributed system, the zone focuses more on physical isolation and fault recovery, then the synchronization flag of the global synchronization control is the zone synchronization flag), the global multi-site pause is enabled, the zone synchronization status is configured to be disabled (the zone synchronization flag is configured to indicate synchronization pause), and peroid update is performed; when initializing the synchronization thread, the zones whose synchronization status is disabled are deleted from the zones that need to be synchronized; subsequently, the synchronization thread that synchronizes with the zone will no longer be enabled, so that the global zone-level synchronization pause control can be implemented, and datalog directly stops updating.

[0074] Reference Figure 5 , Figure 5 This is a partial logic diagram of global level synchronous recovery described in an embodiment of the present application. Figure 5As shown in the figure, global multi-site synchronization recovery is enabled, the zone synchronization status is configured to be enabled (the zone synchronization flag is configured to not indicate synchronization pause), and peroid update is performed; when initializing the synchronization thread, the zones whose synchronization status is enabled are added to the zones that need to be synchronized; subsequently, the synchronization threads corresponding to the zones that are enabled will be traversed. Because the datalogs between the two zones are inconsistent during the pause, synchronization will be automatically performed based on the information recorded in the datalog after the synchronization thread is enabled again.

[0075] Exemplarily, in one embodiment of the present application, the main steps of global level synchronization suspension include: Step 1. In the global control of multiple sites, add a synchronization flag for global pause and resume control; Step 2: After setting the global pause, set the synchronization status in the zone group to disabled, and execute peroidupdate to update the zone related information; Step 3: Update the zone and determine the synchronization status of the current zone when parsing the synchronization policy. If it is disabled, remove the relevant information of the zone from the synchronization zones; if it is enabled, add it to the synchronization zones; step4. According to the relevant operations in step3, choose whether to start the synchronization thread of the corresponding zone.

[0076] Reference Figure 6 , Figure 6 Schematic diagram of the system structure of the multi-site active-active feature in one embodiment of the present application. Figure 6 As shown, the system includes a first site and a second site. Zones are configured in the first site and the second site. Multiple buckets are stored in the zones. Data synchronization operations between the sites are determined based on datalog.

[0077] Reference Figure 7 , Figure 7 Schematic diagram of the framework of the data synchronization control device in one embodiment of the present application. Figure 7 As shown, in one embodiment of the present application, based on the same or similar technical concept as the above embodiment, a data synchronization control device 700 is provided, including: A first determination module 701 is used to determine whether to perform a data synchronization operation according to a first log file of a first site; The acquisition module 702 is used to call the synchronization thread to acquire the synchronization pipeline information of the bucket to be synchronized in the first site when it is determined to perform the data synchronization operation, wherein the synchronization pipeline information includes the identifiers of the target bucket and the source bucket; A second determination module 703 is used to determine whether the bucket synchronization flag bits of the target bucket and the source bucket indicate synchronization pause according to the identifier of the target bucket and the identifier of the source bucket; The first control module 704 is used to control the synchronization process to skip the synchronization operation on the bucket to be synchronized when the synchronization flag bit of any bucket in the target bucket and the source bucket indicates synchronization suspension; The second control module 705 is used to control the synchronization process to perform a synchronization operation on the bucket to be synchronized when the bucket synchronization flag bits of the target bucket and the source bucket do not indicate synchronization pause; The updating module 706 is used to update the first log file.

[0078] In this embodiment, according to the first log file of the first site, it is determined whether to perform the data synchronization operation; in the case of determining to perform the data synchronization operation, the synchronization thread is called to obtain the synchronization pipeline information of the bucket to be synchronized in the first site, and the synchronization pipeline information includes the identifiers of the target bucket and the source bucket; according to the identifier of the target bucket and the identifier of the source bucket, it is determined whether the barrel synchronization flag of the target bucket and the source bucket indicates synchronization suspension; in the case that the barrel synchronization flag of any one of the target bucket and the source bucket indicates synchronization suspension, the synchronization process is controlled to skip the synchronization operation on the barrel to be synchronized; in the case that the barrel synchronization flag of the target bucket and the source bucket does not indicate synchronization suspension, the synchronization process is controlled to perform the synchronization operation on the barrel to be synchronized; and the first log file is updated. Through the barrel synchronization flag of each barrel, data synchronization control of a single barrel can be realized, and after skipping the synchronization operation on the barrel with synchronization suspension, the synchronization operation of other barrels is not affected, and the first log file is still updated, so that the update status between the sites remains consistent, so that the data synchronization control at the barrel level is realized, the system can be controlled in a fine-grained manner, and the resource allocation of the system can be flexibly adjusted.

[0079] In the present application, the description of the device embodiment is relatively short, and specific details can be referred to the above method embodiment, which will not be repeated here.

[0080] In one embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data synchronization control method when executing the computer program.

[0081] In one embodiment of the present application, a non-volatile storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data synchronization control method as described in the above embodiment is implemented.

[0082] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0083] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0084] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0087] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0088] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0089] The above is a detailed introduction to a data synchronization control method, device, electronic device and medium provided. Specific examples are used in this article to illustrate the principles and embodiments of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific embodiments and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data synchronization control method, characterized in that: The method comprises: Determining whether to perform a data synchronization operation according to a first log file of the first site; In the case of determining to perform a data synchronization operation, calling a synchronization thread to obtain synchronization pipeline information of a bucket to be synchronized in the first site, wherein the synchronization pipeline information includes identifiers of a target bucket and a source bucket; Determine, according to the identifier of the target bucket and the identifier of the source bucket, whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization pause; When the synchronization flag bit of any bucket in the target bucket and the source bucket indicates synchronization suspension, the synchronization process is controlled to skip the synchronization operation on the bucket to be synchronized; When the bucket synchronization flags of the target bucket and the source bucket do not indicate synchronization pause, controlling the synchronization process to perform a synchronization operation on the bucket to be synchronized; The first log file is updated.

2. The method according to claim 1, characterized in that The determining whether to perform a data synchronization operation according to the first log file of the first site includes: comparing the first log file with a second log file of a second site; When the value of the update mark of the second log file is higher than the value of the update mark of the first log file, it is determined that the first site needs to perform a data synchronization operation.

3. The method according to claim 1, characterized in that In the case of determining to perform a data synchronization operation, calling a synchronization thread to obtain synchronization pipeline information of a bucket to be synchronized in the first site includes: In the case of determining to perform a data synchronization operation, parsing the first log file to determine an identifier of a bucket to be synchronized; According to the identifier of the bucket to be synchronized, the synchronization pipeline information of the bucket to be synchronized is obtained.

4. The method according to claim 1, characterized in that: The determining, according to the identifier of the target bucket and the identifier of the source bucket, whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization suspension includes: According to the identifier of the target bucket and the identifier of the source bucket, acquiring the attribute information of the target bucket and the attribute information of the source bucket; the attribute information of the bucket includes a bucket synchronization flag; According to the value of the bucket synchronization flag bit, it is determined whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization pause.

5. The method according to claim 1, characterized in that The method further comprises: In the case where the bucket synchronization flag of the bucket to be synchronized is adjusted from indicating synchronization pause to not indicating synchronization pause, the first site sends a recovery request to the second site; the second site includes a source bucket corresponding to the bucket to be synchronized; the recovery request is used to update the value of the update identifier of the second log file of the second site, so that the value of the update identifier of the second log file is higher than the value of the update identifier of the first log file; Determine, according to the updated second log file, a mapping relationship between the second log and a third log in the second site; Determining, according to the mapping relationship, the incremental data of the second site stored in the third log file; Determine the target data in the incremental data according to the synchronization channel information of the bucket to be synchronized; the target data is: the operation data executed by the source bucket corresponding to the bucket to be synchronized during the most recent suspension of the synchronization operation of the bucket to be synchronized; The synchronization thread is called to synchronize the target data into the bucket to be synchronized.

6. The method according to any one of claims 1 to 5, characterized in that: Before determining whether to perform a data synchronization operation according to the first log file of the first site, the method further includes: Detecting whether a cluster synchronization flag of the cluster in the first site indicates synchronization suspension; When the cluster synchronization flag indicates synchronization suspension, the synchronization thread corresponding to the cluster whose cluster synchronization flag indicates synchronization suspension is disabled; Stop updating the first log file.

7. The method according to claim 6, characterized in that The method further comprises: When the cluster synchronization flag of the cluster is adjusted to indicate that the synchronization is not suspended, triggering the first site to perform an update operation on the cluster configuration information; According to the updated cluster configuration information, the cluster synchronization flag is enabled to indicate the synchronization thread corresponding to the unsynchronized paused cluster.

8. A data synchronization control device, characterized in that: The device comprises: A first determination module, used to determine whether to perform a data synchronization operation according to a first log file of a first site; An acquisition module, configured to, when determining to perform a data synchronization operation, call a synchronization thread to acquire synchronization pipeline information of a bucket to be synchronized in the first site, wherein the synchronization pipeline information includes identifiers of a target bucket and a source bucket; A second determination module is used to determine whether the bucket synchronization flag bits of the target bucket and the source bucket respectively indicate synchronization suspension according to the identifier of the target bucket and the identifier of the source bucket; A first control module is used to control the synchronization process to skip the synchronization operation on the bucket to be synchronized when the synchronization flag bit of any bucket in the target bucket and the source bucket indicates synchronization suspension; A second control module is used to control the synchronization process to perform a synchronization operation on the bucket to be synchronized when the bucket synchronization flag bits of the target bucket and the source bucket do not indicate synchronization pause; An updating module is used to update the first log file.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data synchronization control method as described in any one of claims 1-7 is implemented.

10. A non-volatile storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the data synchronization control method as described in any one of claims 1-7 is implemented.

Images