Data synchronization method and device, electronic equipment and storage medium

Abstract

The invention relates to a data synchronization method and device, electronic equipment and a storage medium. The data synchronization method comprises the following steps: acquiring to-be-synchronized data, wherein the to-be-synchronized data corresponds to a plurality of partitions; a synchronization task is obtained based on the to-be-synchronized data and a task model, a plurality of preset parameter items are recorded in the task model, the task model is used for conducting parameter value mapping on the preset parameter items based on the to-be-synchronized data to generate a synchronization task corresponding to the to-be-synchronized data, and the preset parameter items at least comprise partition type parameter items; the data synchronization module is used for recording a data synchronization mode of to-be-synchronized data as specified partition synchronization or overall data synchronization; and determining a currently executed synchronization task in the synchronization tasks, and determining a to-be-synchronized partition in the plurality of partitions for the currently executed synchronization task based on a data synchronization mode recorded by the partition type parameter item, so as to synchronize data in the to-be-synchronized partition to the target cluster. By means of the data synchronization method and device, data synchronization is conducted according to the partition granularity, and the data synchronization efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] With the widespread application of distributed database systems, the synchronization of table data files between database clusters has become increasingly important. There is a need for table data synchronization between different cluster environments, such as data synchronization and migration between the database primary cluster and backup cluster, the official online environment and the test and development environment.

[0003] In the related art, data synchronization between different clusters is usually performed through cluster tables or database directories. The synchronization granularity is coarse, the application is inflexible, the synchronization efficiency is low, and network bandwidth and resource problems are prone to occur. Therefore, the development of a high-performance distributed synchronization method has become an urgent problem to be solved. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data synchronization method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a data synchronization method is provided, comprising obtaining data to be synchronized, wherein the data to be synchronized corresponds to multiple partitions; obtaining a synchronization task based on the data to be synchronized and a task model, wherein the task model records multiple preset parameter items and is used to map parameter values ​​of the preset parameter items based on the data to be synchronized to generate a synchronization task corresponding to the data to be synchronized, wherein the preset parameter items include at least a partition type parameter item, and the partition type parameter item is used to record whether a data synchronization method for the data to be synchronized is specified partition synchronization or overall data synchronization; determining a currently executed synchronization task in the synchronization task, and determining a partition to be synchronized among the multiple partitions based on the data synchronization method recorded in the partition type parameter item for the currently executed synchronization task, so as to synchronize the data in the partition to be synchronized to a target cluster.

[0006] In one embodiment, the data synchronization method recorded in the partition type parameter item is used to determine the partition to be synchronized among the multiple partitions, including: in response to the data synchronization method recorded in the partition type parameter item being designated partition synchronization, based on the start partition parameter item and the end partition parameter item recorded in the task model, determining the start partition and the end partition, and determining the start partition, the end partition, and the partition between the start partition and the end partition as the partitions to be synchronized.

[0007] In one embodiment, the data synchronization method recorded by the partition type parameter item is used to determine the partition to be synchronized among the multiple partitions, including: in response to the data synchronization method recorded by the partition type parameter item being overall data synchronization, determining the time partition among the multiple partitions; if there is a time partition among the multiple partitions, determining the start time partition, the end time partition, and the partition between the start time partition and the end time partition in the time partition as the partition to be synchronized.

[0008] In one implementation, synchronizing the data in the partition to be synchronized to the target cluster includes: obtaining a batch partition number, where the batch partition number is used to represent a maximum number of partitions synchronized in a single time;

[0009] In response to the number of partitions to be synchronized being greater than the number of batch partitions, the partitions to be synchronized are divided into multiple batches according to the number of batch partitions, wherein the number of partitions in each of the multiple batches is less than or equal to the maximum number of partitions; the multiple batches are executed in parallel to synchronize the data to be synchronized to the target cluster.

[0010] In one implementation, determining the currently executing synchronization task in the synchronization task includes: determining the currently executing synchronization task in the synchronization task based on at least one of the following: the maximum number of concurrency supported by the target cluster and the number of currently running tasks; the task priority of the synchronization task; the source cluster label of the synchronization task, the source cluster label is used to match the corresponding target cluster; the currently running number of tasks and the load condition of the execution node in the target cluster.

[0011] According to a second aspect of an embodiment of the present disclosure, a data synchronization device is provided, comprising an acquisition unit for acquiring data to be synchronized, wherein the data to be synchronized corresponds to multiple partitions; a processing unit for obtaining a synchronization task based on the data to be synchronized and a task model, wherein the task model records multiple preset parameter items, and is used to map parameter values ​​of the preset parameter items based on the data to be synchronized to generate a synchronization task corresponding to the data to be synchronized, wherein the preset parameter items include at least a partition type parameter item, and the partition type parameter item is used to record whether a data synchronization method for the data to be synchronized is a specified partition synchronization or an overall data synchronization; a synchronization unit for determining a currently executed synchronization task in the synchronization task, and determining a partition to be synchronized among the multiple partitions based on the data synchronization method recorded in the partition type parameter item for the currently executed synchronization task, so as to synchronize the data in the partition to be synchronized to a target cluster.

[0012] In one embodiment, the synchronization unit determines the partition to be synchronized among the multiple partitions based on the data synchronization method recorded by the partition type parameter item in the following manner: in response to the data synchronization method recorded by the partition type parameter item being specified partition synchronization, based on the start partition parameter item and the end partition parameter item recorded in the task model, the start partition and the end partition are determined, and the start partition, the end partition, and the partition between the start partition and the end partition are determined as the partitions to be synchronized.

[0013] In one embodiment, the synchronization unit determines the partition to be synchronized among the multiple partitions based on the data synchronization mode recorded by the partition type parameter item in the following manner: in response to the data synchronization mode recorded by the partition type parameter item being overall data synchronization, a time partition is determined among the multiple partitions; if there is a time partition among the multiple partitions, the start time partition, the end time partition, and the partition between the start time partition and the end time partition in the time partition are determined as the partition to be synchronized.

[0014] In one embodiment, the synchronization unit synchronizes the data in the partitions to be synchronized to the target cluster in the following manner: obtaining the number of batch partitions, where the number of batch partitions is used to represent the maximum number of partitions for a single synchronization; in response to the number of partitions to be synchronized being greater than the number of batch partitions, dividing the partitions to be synchronized into multiple batches according to the number of batch partitions, wherein the number of partitions in each of the multiple batches is less than or equal to the maximum number of partitions; and executing the multiple batches in parallel to synchronize the data to be synchronized to the target cluster.

[0015] In one embodiment, the synchronization unit determines the currently executed synchronization task in the synchronization task in the following manner: the currently executed synchronization task is determined in the synchronization task based on at least one of the following: the maximum number of concurrency supported by the target cluster and the number of currently running tasks; the task priority of the synchronization task; the source cluster label of the synchronization task, the source cluster label is used to match the corresponding target cluster; the currently running number of tasks and the load status of the execution node in the target cluster.

[0016] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the data synchronization method described in the first aspect or any one of the implementations of the first aspect.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data synchronization method described in the first aspect or any one of the embodiments of the first aspect.

[0018] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: obtaining the data to be synchronized, and obtaining a synchronization task based on the data to be synchronized and the task model. Among them, the data to be synchronized corresponds to multiple partitions, and the task model includes preset parameter items, and the preset parameter items include partition type parameter items for recording the data synchronization method of the data to be synchronized for the specified partition synchronization or the overall data synchronization. Based on the specified partition synchronization or the overall data synchronization, the partition to be synchronized is determined among the multiple partitions to synchronize the data in the partition to be synchronized to the target cluster, so as to realize data synchronization at the partition granularity, thereby improving the efficiency of data synchronization.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] Figure 1 The present invention is a flowchart of a data synchronization method according to an exemplary embodiment.

[0022] Figure 2 The present invention is a flowchart showing a method for determining partitions to be synchronized according to an exemplary embodiment.

[0023] Figure 3 The present invention is a flowchart of a data synchronization method according to an exemplary embodiment.

[0024] Figure 4 The present invention is a flowchart of a data synchronization method according to an exemplary embodiment.

[0025] Figure 5 It is a schematic diagram showing a data synchronization architecture according to an exemplary embodiment.

[0026] Figure 6 The invention is a block diagram of a data synchronization device according to an exemplary embodiment.

[0027] Figure 7 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0029] As a data warehouse tool based on the distributed computing platform (Hadoop), Hive usually occupies a core position in data warehouse construction. Therefore, Hadoop's cross-cluster data synchronization mainly solves the synchronization problem of Hive data in different Hadoop clusters. In related technologies, cross-cluster data synchronization is mainly synchronized by cluster tables or database directories through the Distributed Copy (DistCp) tool. The synchronization granularity is coarse, the application is inflexible, and the synchronization efficiency is low. In the process of data synchronization, manual intervention and maintenance are usually required to deal with problems such as network bandwidth overlimit and resources in the data synchronization process, which requires a lot of manpower maintenance costs and is inefficient.

[0030] In view of this, the present disclosure proposes a data synchronization method, which determines different data synchronization methods for the data to be synchronized, and synchronizes the data in the partition to be synchronized to the target cluster according to the specified partition synchronization or the overall data synchronization, thereby improving the efficiency of data synchronization and enabling intelligent management and automatic execution of the data synchronization process.

[0031] Figure 1 FIG. 1 is a flow chart of a data synchronization method according to an exemplary embodiment. Figure 1 As shown, the method includes steps S11 to S13.

[0032] In step S11, the data to be synchronized is obtained, and the data to be synchronized corresponds to multiple partitions.

[0033] In the embodiment of the present disclosure, the multiple partitions corresponding to the data to be synchronized may be time partitions. The data to be synchronized corresponding to the multiple partitions may be understood as being able to obtain the data to be synchronized in the partition dimension based on the partition dimension.

[0034] In step S12, a synchronization task is obtained based on the data to be synchronized and the task model. The task model records a plurality of preset parameter items and is used to map parameter values ​​of the preset parameter items based on the data to be synchronized to generate a synchronization task corresponding to the data to be synchronized.

[0035] In the embodiment of the present disclosure, the preset parameter items include at least a partition type parameter item, wherein the partition type parameter item includes the synchronization of the specified partition and the overall data synchronization, and is used to record the method of data synchronization for the data to be synchronized, for example, the partition type parameter item may be the data synchronization method of the specified partition synchronization or may be the data synchronization method of the specified partition synchronization. It can be understood that the synchronization of the specified partition and the overall data synchronization are the data synchronization methods, if the partition type parameter item records the synchronization of the specified partition, then the synchronization can be performed through the data in the specified partition, and if the partition type parameter item records the synchronization of the overall data, then the overall data can be further judged and synchronized.

[0036] In the exemplary embodiment of the present disclosure, the synchronization task is obtained based on the data to be synchronized and the preset parameter items in the task model, and the preset parameter items can be mapped to parameter values ​​through the data to be synchronized to generate a synchronization task corresponding to the data to be synchronized. The preset parameter items at least include a partition type parameter item, and may also include other parameter items as shown in Table 1.

[0037] Table 1

[0038]

[0039] Exemplarily, the parameter value mapping of the preset parameter items based on the data to be synchronized to generate the synchronization task corresponding to the data to be synchronized may be the example data in Table 2.

[0040] Table 2

[0041]

[0042]

[0043] In the exemplary embodiment of the present disclosure, the synchronization source cluster can be understood as the starting point of data synchronization, and the synchronization target cluster can be understood as the end point of data synchronization. The database where the copy source is located is the database where the data to be synchronized is located, and the physical table where the copy source is located is the physical table where the data to be synchronized is located. The data synchronization direction is the flow direction of data synchronization, that is, the synchronization source cluster to the target cluster. The starting partition is the partition and data range specified to start synchronization during the data synchronization process, and the ending partition is the partition specified to end synchronization during the data synchronization process. The starting partition and the ending partition can be the partitions specified after the user determines the partition type parameter item. The scheduling mode includes periodic timing scheduling and instant scheduling. Periodic timing scheduling refers to configuring a fixed periodic scheduling time for task execution, and triggering the creation and execution of the task after the time is reached. For example, the timing rule of the task configuration can be executed at 1 a.m. every day. Instant scheduling refers to external triggering through a request, and the task creation is executed after the trigger. Scheduling timing rule refers to the periodic time rule of timing scheduling. The partition type parameter item specifies partition synchronization. Based on the partition type parameter item, the synchronization mode of data synchronization can be determined. The synchronization status refers to the current status of the data synchronization task, which can include pending, running, failed, paused, and successful. The maximum bandwidth quota refers to the maximum network bandwidth allowed to be used during data synchronization, and the creation time refers to the creation time of the data synchronization task or configuration. This sample data is for illustrative purposes only and is not intended to be a specific limitation.

[0044] In step S13, the currently executed synchronization task is determined in the synchronization task, and the partition to be synchronized is determined in multiple partitions based on the data synchronization mode recorded in the partition type parameter item for the currently executed synchronization task, so as to synchronize the data in the partition to be synchronized to the target cluster.

[0045] In the disclosed embodiment, the data synchronization process may include multiple synchronization tasks. It is necessary to determine the currently executed synchronization task among the multiple synchronization tasks, and synchronize the data to be synchronized to the target cluster for the currently executed synchronization task.

[0046] In the disclosed embodiment, the synchronization task currently being executed is determined, and the data to be synchronized is synchronized to the target cluster. The data to be synchronized is the data in the partition to be synchronized. Since the synchronization task currently being executed includes a partition type parameter item, based on the data synchronization method recorded in the partition type parameter item, the partition to be synchronized can be determined from multiple partitions, and the data in the partition to be synchronized can be synchronized to the target cluster.

[0047] In the disclosed embodiment, after designing the task model, the task model can be persisted in a database to facilitate obtaining the data to be synchronized, and the data to be synchronized can be directly mapped to the task model to obtain the synchronization task.

[0048] In the disclosed embodiment, a task query interface may be provided to the requester, so that the requester can obtain the synchronization status of the current synchronization task, and return information such as the synchronization status to the requester.

[0049] In the disclosed embodiment, when data is synchronized in a production environment and in different computer room clusters, the network bandwidth flow needs to be monitored due to the limited bandwidth between computer rooms. If the bandwidth usage exceeds the set threshold, an alarm will be triggered and corresponding processing will be performed.

[0050] In the disclosed embodiment, based on the preset task model, after obtaining the data to be synchronized, the synchronization task corresponding to the data to be synchronized is generated according to the task model, and then the data synchronization method is determined based on the partition type parameter item in the task model. At the same time, based on monitoring the network bandwidth flow, it is helpful to effectively manage network resources. In a high-flow environment, it can prevent the synchronization task from occupying too much bandwidth, thereby affecting the performance of other applications. And by reasonably setting the threshold, selecting the appropriate monitoring tool and alarm mechanism, the stability and efficiency of the data synchronization process are ensured.

[0051] In the disclosed embodiment, based on the data synchronization method recorded by the partition type parameter item in the synchronization task, the partition to be synchronized can be determined from multiple partitions. Among them, if the partition type parameter item of the synchronization task is the specified partition synchronization, that is, the data synchronization method recorded by the partition type parameter item is the specified partition synchronization, then based on the start partition parameter item and the end partition parameter item in the task model, the start partition and the end partition are determined. It can be understood that to determine that the partition type parameter item is the specified partition synchronization, the user needs to specify the start partition and the end partition, so as to obtain the start partition and the end partition based on the start partition parameter item and the end partition parameter item.

[0052] In the disclosed embodiment, the starting partition and the ending partition are determined, and the starting partition, the ending partition, and the partition between the starting partition and the ending partition are determined as the partitions to be synchronized. It should be noted that in the process of data synchronization, the actual partition can be the partition of the execution progress, the ending partition, and the partition between the partition of the execution progress and the ending partition. For example, the starting partition is 20200101 and the ending partition is 20200301, then the partitions to be synchronized are 20200101, 20200201 and 20200301. In the process of data synchronization, if the current partition 20200101 has been synchronized, the current starting partition is 20200201 and the ending partition is 20200301.

[0053] It can be understood that the data to be synchronized is obtained, and a synchronization task is generated based on the data to be synchronized and the task model. Since the task model includes a partition type parameter item, the user can determine whether the partition type parameter item is a specified partition synchronization or an overall data synchronization. If the partition type parameter item is a specified partition synchronization, the user specifies the starting partition and the ending partition through the starting partition parameter item and the ending partition parameter item. The starting partition and the ending partition are determined, and the partition to be synchronized is obtained based on the starting partition, the ending partition, and the partition between the starting partition and the ending partition, and the data in the partition to be synchronized is used as the data to be synchronized corresponding to the currently executed synchronization task. It can be understood that the acquired data to be synchronized and the data to be synchronized in the partition to be synchronized contain the same data, only the dimensions are different.

[0054] In the disclosed embodiment, data is synchronized according to the designated partitions, so that the data to be synchronized is synchronized according to the partition granularity, improving the synchronization efficiency of data synchronization, and providing users with a designated partition synchronization option to avoid consuming too many data resources in the case of overall data synchronization.

[0055] In the embodiment of the present disclosure, if the partition type parameter item is overall data synchronization, it is necessary to determine the time partition in the data to be synchronized, and synchronize based on the data in the time partition.

[0056] Figure 2FIG. 1 is a flow chart showing a method for determining a partition to be synchronized according to an exemplary embodiment. Figure 2 As shown, the method includes steps S21 to S22.

[0057] In step S21, in response to the data synchronization mode recorded in the partition type parameter item being overall data synchronization, a time partition is determined among the multiple partitions.

[0058] In the embodiment of the present disclosure, the data synchronization mode corresponding to the currently executed synchronization task is overall data synchronization, and the time partition corresponding to the data to be synchronized can be determined from the source data table of the data to be synchronized. The time partition can be all partitions between the earliest historical partition and the latest current partition in the source data table of the data to be synchronized, or it can be all partitions between the execution progress partition and the end partition during the data synchronization execution process.

[0059] In step S22, if there is a time partition among the multiple partitions, the start time partition, the end time partition, and the partition between the start time partition and the end time partition among the time partitions are determined as the partitions to be synchronized.

[0060] In the embodiment of the present disclosure, if it is determined that there is a time partition in the source data table of the data to be synchronized, the partition to be synchronized is determined based on the start time partition, the end time partition, and the partition between the start time partition and the end time partition. Among them, the start time partition can be understood as the earliest partition in the history of the source data table of the data to be synchronized, and the end time partition can be understood as the latest partition in the source data table of the data to be synchronized. If the earliest partition in history has been synchronized during the data synchronization execution process, the start time partition can be sorted in the front of the partition being executed.

[0061] In the embodiment of the present disclosure, in response to the data synchronization mode recorded by the partition type parameter item being overall data synchronization, and there being no time partition in the data to be synchronized, the data to be synchronized is not divided into batches, and the parent directory of the source cluster table of the data to be synchronized is synchronized to the designated directory of the target cluster to complete the data synchronization. If the synchronization task is completed normally, the task status in the update database is completed; if there is an exception in the synchronization task, the task status in the update database is failed.

[0062] In the disclosed embodiment, the executor provides an interface service for task interruption, which is responsible for closing the task that is executing synchronization and updating the task status to pause.

[0063] In the disclosed embodiment, before data synchronization, it is necessary to confirm whether the target table exists in the target cluster. If not, the table structure is exported from the source cluster and the synchronized target table is created in the target cluster, so that the data can be mapped to the target table after being synchronized to the target cluster.

[0064] In the disclosed embodiment, if the partition type parameter item is overall data synchronization, based on the data synchronization method of overall data synchronization, the time partition in the data to be synchronized, that is, the earliest partition in history and the latest partition at present, is determined, and the data to be synchronized is synchronized as the start time partition and the end time partition. If there is no time partition in the data to be synchronized, the overall data can also be synchronized to the target cluster according to the granularity of the table. That is, the diversity of the overall data synchronization in the data synchronization method is provided, and the method of synchronizing the overall data according to the partition granularity is guaranteed, which improves the synchronization efficiency of the overall data during synchronization.

[0065] In the embodiment of the present disclosure, the partition to be synchronized is determined based on the partition type parameter item, and then the data in the partition to be synchronized can be synchronized to the target cluster. Figure 3 FIG. 1 is a flow chart of a data synchronization method according to an exemplary embodiment. Figure 3 As shown, the method includes steps S31 to S33.

[0066] In step S31, the number of batch partitions is obtained.

[0067] In the disclosed embodiment, the number of batch partitions is used to characterize the maximum number of partitions for a single synchronization. For example, three batches are pre-set, and each batch can accommodate two partitions, that is, the maximum number of partitions for a single synchronization of each batch is two, that is, the number of batch partitions is two. The above example is only an illustrative description, and the number of batch partitions is not specifically limited.

[0068] In step S32, in response to the number of partitions to be synchronized being greater than the number of batch partitions, the partitions to be synchronized are divided into a plurality of batches according to the number of batch partitions.

[0069] In the embodiment of the present disclosure, if the number of partitions to be synchronized is greater than the number of batch partitions, the partitions to be synchronized can be divided according to the number of batch partitions to obtain multiple batches after division. It can be understood that the number of partitions in each batch of the multiple batches after division is less than or equal to the maximum number of partitions in the batch partition number.

[0070] In the exemplary embodiment of the present disclosure, if the number of partitions to be synchronized is 6 and the number of batch partitions is 2, that is, the maximum number of partitions is 2, and the number of partitions to be synchronized is greater than the number of batch partitions, then the number of partitions to be synchronized is divided according to the number of batch partitions to obtain 3 batches, each of which includes 2 partitions. Or if the number of partitions to be synchronized is 7 and the number of batch partitions is 2, then it is divided into 4 batches, of which 3 batches include 2 partitions and 1 batch includes 1 partition. The above data is only for illustrative purposes and is not specifically limited.

[0071] It should be noted that during the data synchronization process, the data synchronization progress can be updated in real time. For each partition of the synchronization task that has been processed, the synchronization progress of the task will be updated in real time. If the partition processing is completed, the synchronization progress of the task will be updated to the current partition. If the synchronization task is completed normally, the task status in the database is updated to completed. If there is an exception in the synchronization task, the task status in the database is updated to failed.

[0072] In step S33, multiple batches are executed in parallel to synchronize the data to be synchronized to the target cluster.

[0073] In the disclosed embodiment, the partitions to be synchronized are divided according to the number of batch partitions to obtain multiple batches after division. Each batch is executed in parallel. If each batch includes multiple partitions, the multiple partitions in a batch are executed serially. Based on the joint execution of serial and parallel, the data to be synchronized is synchronized to the target cluster.

[0074] In the embodiment of the present disclosure, before data synchronization is performed, it is confirmed whether the target table exists in the target cluster. If not, the table structure needs to be exported from the source cluster, and the synchronized target table is created in the target cluster based on the table structure. The target table can be understood as the table corresponding to the data to be synchronized.

[0075] In the embodiment of the present disclosure, after the data to be synchronized is synchronized to the target cluster based on the grouped data sets to be synchronized, the grouped data sets to be synchronized are activated in the target cluster so that the synchronized data can be queried.

[0076] In the disclosed embodiment, data synchronization is performed based on partition granularity and multiple batches are executed in parallel, which can not only ensure that the synchronized data granularity is fine, but also improve the synchronization efficiency of data synchronization through parallel execution, thereby improving the user experience during the data synchronization process.

[0077] In the disclosed embodiment, since there will be multiple synchronization tasks, it is necessary to obtain the currently executed synchronization task from the multiple synchronization tasks through a scheduling strategy, and the currently executed synchronization task can be determined through the maximum concurrency number and the currently running number of tasks supported by the target cluster, the task priority of the synchronization task, and the source cluster label of the synchronization task. The source cluster label is used to match the corresponding target cluster and at least one of the currently running number of tasks and the load condition of the execution node in the target cluster.

[0078] In the disclosed embodiment, the maximum concurrency supported by the target cluster and the maximum concurrency of the target cluster in the number of currently running tasks are the maximum concurrency set for the target cluster. The number of synchronization tasks to be taken out from the task pool is determined based on the number of currently running tasks of the target cluster and the maximum concurrency. Task priority can be understood as if the synchronization task is set with a priority identifier, it is taken out from the task pool in order of priority. Synchronization tasks with higher priorities are executed first. If the user does not specify the task priority of the synchronization task, the default priority of the synchronization task is 0. If the priorities are the same, the synchronization task with an earlier creation time is executed first.

[0079] In the disclosed embodiment, the source cluster label of the synchronization task, wherein the source cluster label is used to match the corresponding target cluster. For example, if the label of the synchronization task is cluster a, the synchronization task is assigned to the execution node of cluster a. The current number of running tasks and the load condition of the execution node in the target cluster can be understood as if there are multiple assignable execution nodes for the synchronization task, it needs to be determined based on the load condition of the execution node and the number of assigned tasks of the current execution node, for example, selecting the execution node with low load and small number of execution tasks to assign tasks in descending order of load and number of execution nodes.

[0080] In the disclosed embodiment, synchronization tasks are acquired based on a scheduling strategy, thereby achieving efficient use of resources, reasonably acquiring synchronization tasks, avoiding high load situations, and thereby improving the synchronization efficiency of synchronization tasks.

[0081] In the disclosed embodiment, before obtaining the synchronization task, it is necessary to deploy nodes based on the scheduler and the task execution module to achieve the distribution and reception of the synchronization task.

[0082] In the disclosed embodiment, a scheduling master node may be set, and the scheduling master node is used to obtain synchronization tasks and distribute synchronization tasks to execution nodes. The scheduling master node is obtained through election. Specifically, it can be understood that the scheduler deploys multiple nodes. When each scheduler is running, the scheduling master node is elected through the framework (Apache Curator). The scheduling master node is obtained after the election, and other nodes are responsible for monitoring the survival status of the master node. When the scheduling master node fails, re-election is performed to ensure the availability of the system.

[0083] In the disclosed embodiment, a newly registered execution node can also be obtained and added to the executable node list. If the execution node exits, the faulty execution node is removed from the executable node list, and the task status assigned to the execution node is queried, and the executing task is marked as a failed state so that other execution nodes can restore the synchronization task. The faulty execution node is logged in to close the running synchronization task.

[0084] In the disclosed embodiment, the scheduling master node obtains the synchronization task based on the scheduling strategy, and distributes the synchronization task to the task node of the task execution module. Among them, the task node is deployed in advance, and multiple task nodes can be deployed. The task node is used to execute multiple synchronization tasks in a distributed manner. And the synchronization task distributed by the scheduling master node can be received by requesting a path to receive a security identifier (receive / {sid} / ).

[0085] In the disclosed embodiment, a distributed synchronization execution method is provided through node registration, scheduling the main node to distribute synchronization tasks, and task nodes to receive synchronization tasks, thereby improving the efficiency of data synchronization. During business peak periods, the system's throughput capacity can be improved and the operation can be efficient through rapid expansion or reduction of nodes.

[0086] In the embodiments of the present disclosure, since the synchronization tasks are running in a multi-task sharing environment, it is necessary to ensure the stable operation of the synchronization tasks and other types of tasks. Therefore, a regulation strategy can be designed based on factors such as execution time, network environment and resource conditions, and real-time monitoring of data synchronization tasks is performed based on the regulation strategy.

[0087] In the disclosed embodiment, there may be the following strategies: flow limiting strategy. If the machine network card traffic reaches or exceeds the preset threshold or other abnormalities, the currently executing synchronization task will be interrupted and marked as suspended, and new tasks will be blocked from being submitted so that they can be retried in idle time. Synchronous task execution period strategy. The executable period of the synchronization task is predefined. The execution time of the synchronization task will be staggered with the intensive reading and writing period of other tasks. No new tasks will be submitted during the non-executable period. Tasks that have not been completed during the executable period will continue to be executed within the timeout period. If they have not been completed within the timeout period, they will be marked as suspended so that they can be retried in idle time. Data synchronization task concurrency limit strategy. The number of tasks running at the same time is limited. If the number exceeds, no new tasks will be submitted. Resource limit strategy. The resource size of a single task and the total resource usage size of the running tasks are limited. The total resource size exceeds the threshold. No new tasks will be submitted. Manual intervention is required for the resource excess of a single task. Task timeout killing strategy. If the task execution time exceeds the maximum task running time, such as setting the threshold to 120 minutes, the task will be suspended if it runs for more than 120 minutes. To prevent tasks from running on slow nodes or data skew, which causes tasks to run for a long time and occupy resources, the system will indicate the task as abnormal, and the task can be retried through manual intervention and analysis. Task retry strategy: tasks marked as failed will be re-executed within the retry limit. If the retry limit is exceeded, recovery will be required under manual intervention.

[0088] In the disclosed embodiment, for situations such as current limiting strategies, the scheduler needs to interrupt some of the tasks that are currently running. The scheduler obtains the executor of the specific task to be interrupted and notifies the executor to interrupt the task through the executor's interface. After the recovery conditions of the interrupted task are met, such as network bandwidth recovery or resource recovery, the specific task is recovered by updating the task's status to pending execution. If the bandwidth usage exceeds the threshold, an alarm will be triggered, and the scheduler will prevent the execution of new synchronization tasks and will not assign new tasks to the executor for execution. It will also close some of the tasks that are currently being executed until the bandwidth recovers below the threshold, and compare the status of the closed task to failure so that the task can be retried and recovered.

[0089] In the disclosed embodiment, intelligent management and control can be automatically executed during the data synchronization process based on the control strategy, thereby reducing maintenance costs and improving the execution efficiency of data synchronization.

[0090] In the present disclosure, Figure 4 The data synchronization method is described. Figure 4 FIG. 1 is a flow chart of a data synchronization method according to an exemplary embodiment. Figure 4 As shown, it includes an interaction module, a scheduler module, and an execution module. In the interaction module, a task model is pre-designed. After obtaining the user's synchronization request, the interaction module generates a synchronization task based on the task model and synchronization information, and persists the task model task to the database. The interaction module can also provide an interface for user task status query, network bandwidth monitoring alarm, and task failure monitoring alarm.

[0091] In the disclosed embodiment, the scheduler module needs to elect a master node, that is, to elect a master node among multiple schedulers, and the scheduler that successfully elects the master node can be understood as the master scheduler. The master scheduler obtains the tasks to be synchronized through the scheduler scheduling strategy and distributes the tasks. The data synchronization tasks can also be regulated in real time based on the regulation strategy, and the task status can be updated to the database.

[0092] In the disclosed embodiment, the execution module is used to perform node cluster labeling and node registration, and register the node with Zookeeper. And receive task requests, whose task requests are distributed by the main scheduler in the scheduler module, and perform task execution based on the task request through the DistCp data synchronization method and the metadata data synchronization method such as table and partition, so as to synchronize the data from the cluster source-cluster to the cluster target-cluster. And the interruption of the task execution process in the execution module can be solved through the control strategy of the scheduler module.

[0093] In the embodiments of the present disclosure, further combined Figure 5 The data synchronization method is described. Figure 5FIG. 1 is a schematic diagram showing a data synchronization architecture according to an exemplary embodiment. Figure 5 As shown, it includes an interaction module, a database, a scheduler module and an executor module. The interaction module includes parsing, creating and persisting tasks to be synchronized, creating periodic tasks, monitoring bandwidth thresholds and notifying and alarming for failure of tasks to be synchronized.

[0094] In the disclosed embodiment, the database includes a task table, a log table and other tables. The scheduler module sets the priority management of the tasks to be synchronized, and performs the scheduling of the tasks to be synchronized to the execution nodes, the interruption management of the tasks to be synchronized, and the retry of the tasks to be synchronized. The executor module includes the labeling of the execution node cluster, the execution of the tasks to be synchronized, the logging of the tasks to be synchronized, the state switching, the exception handling and the retry.

[0095] In the exemplary embodiment of the present disclosure, the scheduler module and the executor module are described. Among them, the scheduler that successfully selects the master in the scheduler module obtains the successfully registered execution node and the identifier (Internet Protocol, IP) of the execution node and the service port information from Zookeeper to form a node list. And obtain the newly registered execution node, and add the execution node to the executable node list. The scheduler that successfully selects the master queries the tasks to be synchronized from the database and assigns the tasks to be synchronized to the executable node list. If the execution node exits, the faulty node is removed from the executable node list, and the status of the tasks to be synchronized assigned to the execution node is queried at the same time, and the tasks to be synchronized in execution are marked as failed so that other execution nodes can restore the tasks to be synchronized, and log in to the faulty node to close the client synchronization tasks of the tasks to be synchronized that are still running. The execution node in the executor module is responsible for receiving the tasks sent by the scheduler. Among them, if the execution node is marked with a cluster label, the synchronization task of the cluster corresponding to the cluster table is executed. Multiple execution nodes can be deployed to perform distributed task execution, thereby reducing the pressure on a single node and single point failure, and improving the efficiency of data synchronization.

[0096] In the above embodiments of the present disclosure, user requests are received through an interactive module, tasks to be synchronized are generated, and synchronization status queries are provided. Through the scheduler module, multiple execution nodes are deployed with high availability, and are responsible for assigning tasks to be synchronized to the task execution module. Through the task execution module, multiple nodes are deployed to receive tasks to be synchronized assigned by the scheduler module. Through the above method, a distributed synchronization system is realized, which greatly improves the synchronization efficiency, thereby having distributed scalability and fault tolerance capabilities, and improving the availability and stability of the system. Secondly, according to the partition characteristics of the synchronization table, multiple tasks to be synchronized are split into batches or partition granularities for parallel execution, further improving the efficiency of data synchronization. Based on the collaborative work between the modules, efficient synchronization based on distribution is achieved. And through control strategies, such as task pause reply strategies when the network bandwidth exceeds the threshold, etc., it can be executed during the intelligent management and automation of the data synchronization process, thereby improving the efficiency of data synchronization.

[0097] Based on the same concept, the embodiment of the present disclosure also provides a data synchronization device 100 .

[0098] It is understandable that the data synchronization device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0099] Figure 6 1 is a block diagram of a data synchronization device 100 according to an exemplary embodiment. Figure 6 The device includes an acquisition unit 101, a processing unit 102 and a synchronization unit 103.

[0100] The acquisition unit 101 is used to acquire the data to be synchronized, where the data to be synchronized corresponds to multiple partitions.

[0101] The processing unit 102 is used to obtain a synchronization task based on the data to be synchronized and the task model, wherein a plurality of preset parameter items are recorded in the task model, and is used to map parameter values ​​of the preset parameter items based on the data to be synchronized to generate a synchronization task corresponding to the data to be synchronized, wherein the preset parameter items include at least a partition type parameter item, and the partition type parameter item is used to record whether the data synchronization method for the data to be synchronized is a specified partition synchronization or an overall data synchronization.

[0102] The synchronization unit 103 is used to determine the currently executed synchronization task in the synchronization task, and determine the partition to be synchronized among multiple partitions based on the data synchronization mode recorded in the partition type parameter item for the currently executed synchronization task, so as to synchronize the data in the partition to be synchronized to the target cluster.

[0103] In one embodiment, the synchronization unit 103 determines the partition to be synchronized among multiple partitions based on the data synchronization method recorded by the partition type parameter item in the following manner: in response to the data synchronization method recorded by the partition type parameter item being specified partition synchronization, the start partition and the end partition are determined based on the start partition parameter item and the end partition parameter item recorded in the task model, and the start partition, the end partition, and the partition between the start partition and the end partition are determined as the partitions to be synchronized.

[0104] In one embodiment, the synchronization unit 103 determines the partition to be synchronized among multiple partitions based on the data synchronization method recorded by the partition type parameter item in the following manner: in response to the data synchronization method recorded by the partition type parameter item being overall data synchronization, a time partition is determined among multiple partitions; if there are time partitions among multiple partitions, the start time partition, the end time partition, and the partition between the start time partition and the end time partition in the time partition are determined as the partition to be synchronized.

[0105] In one embodiment, the synchronization unit 103 synchronizes the data in the partitions to be synchronized to the target cluster in the following manner: obtaining the number of batch partitions, where the number of batch partitions is used to represent the maximum number of partitions for a single synchronization; in response to the number of partitions to be synchronized being greater than the number of batch partitions, dividing the partitions to be synchronized into multiple batches according to the number of batch partitions, wherein the number of partitions in each of the multiple batches is less than or equal to the maximum number of partitions; and executing multiple batches in parallel to synchronize the data to be synchronized to the target cluster.

[0106] In one implementation, the synchronization unit 103 determines the currently executed synchronization task in the synchronization task in the following manner: the currently executed synchronization task is determined in the synchronization task based on at least one of the following: the maximum number of concurrency supported by the target cluster and the number of currently running tasks; the task priority of the synchronization task; the source cluster label of the synchronization task, the source cluster label is used to match the corresponding target cluster; the currently running number of tasks and the load status of the execution node in the target cluster.

[0107] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0108] like Figure 7 As shown, Figure 7: is a schematic diagram of an electronic device provided by an embodiment of the present invention. One embodiment of the present invention provides an electronic device 200. Among them, the electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. Among them, the memory 201 is used to store instructions. The processor 202 is used to call the instructions stored in the memory 201 to execute the file merging method of the embodiment of the present invention. Among them, the processor 202 is connected to the memory 201 and the I / O interface 203 respectively, for example, it can be connected through a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program of the file merging method involved in the embodiment of the present invention, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the program stored in the memory 201.

[0109] In the embodiment of the present invention, the processor 202 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 202 can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0110] The memory 201 in the embodiment of the present invention may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD).

[0111] In the embodiment of the present invention, the I / O interface 203 may be used to receive input instructions (such as digital or character information, and key signal input related to user settings and function control of the electronic device 200), and may also output various information to the outside (such as images or sounds). In the embodiment of the present invention, the I / O interface 203 may include one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0112] It is to be understood that, although the operations are described in a specific order in the drawings in the embodiments of the present invention, it should not be understood as requiring to perform these operations in the specific order shown or in a serial order, or requiring to perform all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0113] The methods and devices involved in the embodiments of the present invention can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "device" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0114] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

[0115] The foregoing description of the implementation of the present invention has been given for purposes of illustration and description. The foregoing description is not intended to be exhaustive nor to limit the invention to the exact form disclosed, and various variations and modifications are possible in accordance with the above teachings or may be obtained from the practice of the invention. These embodiments are selected and described in order to illustrate the principles of the invention and its practical application, so that those skilled in the art can utilize the invention in various embodiments and with various modifications suitable for the particular use contemplated.

Claims

1. A data synchronization method, characterized in that: include: Acquire data to be synchronized, where the data to be synchronized corresponds to multiple partitions; Based on the data to be synchronized and the task model, a synchronization task is obtained, wherein a plurality of preset parameter items are recorded in the task model, and the task model is used to map parameter values ​​of the preset parameter items based on the data to be synchronized to generate a synchronization task corresponding to the data to be synchronized, wherein the preset parameter items at least include a partition type parameter item, and the partition type parameter item is used to record whether the data synchronization mode of the data to be synchronized is a specified partition synchronization or an overall data synchronization; A currently executed synchronization task is determined in the synchronization task, and a partition to be synchronized is determined in the multiple partitions based on the data synchronization mode recorded in the partition type parameter item for the currently executed synchronization task, so as to synchronize the data in the partition to be synchronized to the target cluster.

2. The method according to claim 1, characterized in that The determining a partition to be synchronized among the multiple partitions based on the data synchronization mode recorded in the partition type parameter item includes: In response to the data synchronization mode recorded in the partition type parameter item being specified partition synchronization, the starting partition and the ending partition are determined based on the starting partition parameter item and the ending partition parameter item recorded in the task model, and the starting partition, the ending partition, and the partition between the starting partition and the ending partition are determined as partitions to be synchronized.

3. The method according to claim 1, characterized in that The determining a partition to be synchronized among the multiple partitions based on the data synchronization mode recorded in the partition type parameter item includes: In response to the data synchronization mode recorded in the partition type parameter item being overall data synchronization, determining a time partition among the multiple partitions; If there is a time partition among the multiple partitions, a start time partition, an end time partition, and a partition between the start time partition and the end time partition among the time partitions are determined as partitions to be synchronized.

4. The method according to any one of claims 1 to 3, characterized in that The step of synchronizing the data in the partition to be synchronized to the target cluster includes: Get the number of batch partitions, which is used to represent the maximum number of partitions synchronized in a single time; In response to the number of the partitions to be synchronized being greater than the number of batch partitions, dividing the partitions to be synchronized into multiple batches according to the number of batch partitions, wherein the number of partitions in each of the multiple batches is less than or equal to the maximum number of partitions; The multiple batches are executed in parallel to synchronize the data to be synchronized to the target cluster.

5. The method according to claim 1, characterized in that The step of determining the currently executed synchronization task in the synchronization task includes: Determining a currently executed synchronization task in the synchronization task based on at least one of the following: The maximum number of concurrent tasks and the number of currently running tasks supported by the target cluster; Task priority of synchronization tasks; The source cluster label of the synchronization task, which is used to match the corresponding target cluster; The current number of running tasks and load of the execution nodes in the target cluster.

6. A data synchronization device, characterized in that: include: An acquisition unit, used for acquiring data to be synchronized, wherein the data to be synchronized corresponds to multiple partitions; A processing unit, configured to obtain a synchronization task based on the data to be synchronized and a task model, wherein the task model records a plurality of preset parameter items, and to perform parameter value mapping on the preset parameter items based on the data to be synchronized to generate a synchronization task corresponding to the data to be synchronized, wherein the preset parameter items at least include a partition type parameter item, and the partition type parameter item is used to record whether the data synchronization mode for the data to be synchronized is a specified partition synchronization or an overall data synchronization; A synchronization unit is used to determine the currently executed synchronization task in the synchronization task, and determine the partition to be synchronized in the multiple partitions based on the data synchronization method recorded in the partition type parameter item for the currently executed synchronization task, so as to synchronize the data in the partition to be synchronized to the target cluster.

7. The device according to claim 6, characterized in that The synchronization unit determines the partition to be synchronized among the multiple partitions based on the data synchronization mode recorded by the partition type parameter item in the following manner: In response to the data synchronization mode recorded in the partition type parameter item being specified partition synchronization, the starting partition and the ending partition are determined based on the starting partition parameter item and the ending partition parameter item recorded in the task model, and the starting partition, the ending partition, and the partition between the starting partition and the ending partition are determined as partitions to be synchronized.

8. The device according to claim 6, characterized in that The synchronization unit determines the partition to be synchronized among the multiple partitions based on the data synchronization mode recorded by the partition type parameter item in the following manner: In response to the data synchronization mode recorded in the partition type parameter item being overall data synchronization, determining a time partition among the multiple partitions; If there is a time partition among the multiple partitions, a start time partition, an end time partition, and a partition between the start time partition and the end time partition among the time partitions are determined as partitions to be synchronized.

9. An electronic device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the data synchronization method described in any one of claims 1 to 5.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data synchronization method according to any one of claims 1 to 5.

Images