A data synchronization method, apparatus, and medium

Abstract

The application relates to the field of databases and discloses a data synchronization method and device and a medium, which comprises the following steps: after detecting completion information of any batch processing script, a target data table corresponding to the completion information is acquired; a target task relationship unidirectionally pointing to the target data table is determined according to a task table dependency graph; whether all target task relationships are completed is judged; if yes, data in the target data table is synchronized to each data warehouse. It can be seen that the technical scheme provided by the application analyzes the task relationship of each target data table, executes data synchronization when all task relationships are completed, does not need to wait for the progress of other data tables in the cluster where the target data table is located, and can realize data synchronization in the batch running process, so that the time occupied by the data processing process is reduced, and the data analysis efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of databases, and in particular to a data synchronization method, apparatus, and medium. Background Technology

[0002] With the development of computer technology, more and more enterprises are choosing to store their data in the form of data centers. An enterprise data center often contains multiple big data clusters and multiple data warehouses. To improve work efficiency, batch processing is typically used to process data, writing the data into corresponding data tables for analysis. Batch processing operations are usually completed in a dedicated batch processing data warehouse. After all batch processing operations are completed, the data needs to be synchronized to other data warehouses in a timely manner for production analysis.

[0003] However, due to the large volume of data to be processed and its multiple data levels, hundreds or even thousands of batch processing scripts are needed to process the data, resulting in a long batch processing time. Waiting for all batch processing steps to complete before synchronizing data between clusters would lead to excessively long data processing times, preventing timely data analysis.

[0004] Therefore, it is evident that providing a more efficient data synchronization method to reduce the time spent on data processing and improve data analysis efficiency is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a data synchronization method, apparatus, and medium to achieve data synchronization during batch processing, thereby reducing the time spent on data processing and improving data analysis efficiency.

[0006] To address the aforementioned technical problems, this application provides a data synchronization method, comprising:

[0007] Once completion information for any batch script is detected, the target data table corresponding to the completion information is obtained.

[0008] The target task relationship that points unidirectionally to the target data table is determined based on the task table dependency graph.

[0009] Determine whether all the aforementioned target task relationships have been completed;

[0010] If all is completed, the data in the target data table will be synchronized to each data warehouse.

[0011] Preferably, before the step of detecting completion information of any batch script, the method further includes:

[0012] Obtain and parse each batch script to obtain the batch processing task, source data table and target data table corresponding to each batch script;

[0013] The source data table and the target data table are used as nodes, and the batch processing tasks between the source data table and the target data table are used as unidirectional task relationships to generate the task table dependency graph.

[0014] Preferably, synchronizing the data in the target data table to each data warehouse includes:

[0015] Generate a data synchronization task based on the target data table;

[0016] The data synchronization task is sent to the cache pool, and the priority of the data synchronization task is calculated.

[0017] The data synchronization task in the cache pool is executed according to the stated priority.

[0018] Preferably, calculating the priority of the data synchronization task includes:

[0019] Obtain the initial priority of the synchronization task and the historical data corresponding to the synchronization task;

[0020] Based on the historical data, predict the estimated resource consumption and estimated execution time of the synchronization task;

[0021] The priority is determined based on the initial priority, the estimated resource consumption, and the estimated execution time.

[0022] Preferably, after the step of determining the priority based on the initial priority, the estimated resource consumption, and the estimated execution time, the method further includes:

[0023] Get the execution information of the currently executing batch script;

[0024] The amount of idle computing resources within a threshold time period is determined based on the execution information and the historical data.

[0025] The priority is adjusted based on the amount of idle computing resources, the estimated resource consumption, and the estimated execution time.

[0026] Preferably, executing the data synchronization task in the cache pool according to the priority includes:

[0027] The tasks to be executed are determined based on the aforementioned priority.

[0028] Obtain the estimated resource consumption of the task to be executed;

[0029] Determine whether the current amount of idle resources is greater than the estimated resource consumption.

[0030] If the amount of resource consumption is greater than the estimated amount, then the task to be executed will be performed.

[0031] Preferably, obtaining the target data table corresponding to the completion information includes:

[0032] The batch script corresponding to the completion information is parsed using the Spark SQL parsing tool to obtain the target data table.

[0033] To address the aforementioned technical problems, this application also provides a data synchronization device comprising:

[0034] The acquisition module is used to acquire the target data table corresponding to the completion information when the completion information of any batch processing script is detected.

[0035] The determination module is used to determine the target task relationship that points unidirectionally to the target data table based on the task table dependency graph;

[0036] The judgment module is used to determine whether all the aforementioned target task relationships have been completed.

[0037] The synchronization module is used to synchronize the data in the target data table to each data warehouse once all operations are completed.

[0038] To address the aforementioned technical problems, this application also provides a data synchronization device, including a memory for storing computer programs;

[0039] A processor is used to implement the steps of the data synchronization method when executing the computer program.

[0040] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data synchronization method described above.

[0041] This application provides a data synchronization method, including: upon detecting the completion information of any batch processing script, obtaining the target data table corresponding to the completion information; determining the target task relationships pointing unidirectionally to the target data table based on the task table dependency graph; determining whether all target task relationships are completed; and if all are completed, synchronizing the data in the target data table to each data warehouse. Therefore, the technical solution provided by this application, by analyzing the task relationships of each target data table, performs data synchronization only when all task relationships are completed, without waiting for the progress of other data tables in the cluster where the target data table resides. Data synchronization can be achieved during batch processing, thereby reducing the time occupied by data processing and improving data analysis efficiency.

[0042] To solve the above-mentioned technical problems, this application also provides a data synchronization device and medium, which correspond to the above method and have the same effect. Attached Figure Description

[0043] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a data synchronization method provided in an embodiment of this application;

[0045] Figure 2 A flowchart illustrating the generation process of the task table dependency graph provided in this application embodiment;

[0046] Figure 3 This is a schematic diagram of the task table dependency graph provided in an embodiment of this application;

[0047] Figure 4 A flowchart illustrating a data synchronization method provided in an embodiment of this application;

[0048] Figure 5 A flowchart illustrating a priority calculation method provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of future idle resources provided in an embodiment of this application;

[0050] Figure 7 This is a diagram illustrating the execution of a synchronous task.

[0051] Figure 8 This is a structural diagram of a data synchronization device provided in an embodiment of this application;

[0052] Figure 9 This is a structural diagram of another data synchronization device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0054] The core of this application is to provide a data synchronization method, device, and medium to achieve data synchronization during batch processing, thereby reducing the time occupied by the data processing process and improving the efficiency of data analysis.

[0055] The data synchronization method provided in this application is applied to servers and is mainly used to synchronize data between various data clusters or data warehouses in an enterprise data center. Currently, there is no perfect solution for automatically synchronizing data between data warehouses in different clusters. The open-source community provides some tools, such as Apache Sqoop, Alibaba's DataX, or the underlying HDFS Distcp, all of which can be used for data synchronization. However, a set of efficient and intelligent solutions has not yet been formed. In addition to tools, the trigger point for data synchronization is also critical. If data synchronization is performed only after all batch processing tasks have been completed, timeliness will be poor, and batch processing may not be completed after working hours, affecting subsequent data analysis. At the same time, excessive data throughput during synchronization will lead to severe network I / O contention, resulting in low synchronization efficiency. When the core cluster completes batch processing and simultaneously releases data for synchronization, resource contention is very severe, leading to I / O congestion and other problems when network I / O and disk I / O are under high load. However, if synchronization is performed immediately after each batch processing task is completed, it cannot be guaranteed that the data in the synchronized table is in its final state, so other batch processing tasks may also insert data into the same table. Multiple synchronizations like this double the I / O; the inability to guarantee the order of multiple tasks also leads to poor data consistency. To address this technical problem, this application provides a data synchronization method that analyzes the task relationships of each target data table. Once all task relationships are completed, data synchronization is performed without waiting for the progress of other data tables in the cluster where the target data table resides. Data synchronization can be achieved during batch processing, thereby reducing the time spent on data processing and improving data analysis efficiency.

[0056] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Figure 1 A flowchart of a data synchronization method provided in an embodiment of this application is shown below. Figure 1 As shown, data synchronization methods include:

[0058] S10: When the completion information of any batch script is detected, obtain the target data table corresponding to the completion information;

[0059] S11: Determine the target task relationship that points unidirectionally to the target data table based on the task table dependency graph.

[0060] In actual production, data in the same table often originates from the sum of multiple different batch processing tasks. When one batch processing task completes, it doesn't necessarily mean all the data in the target table has been generated; it's necessary to confirm that other tasks running on the same target table have also completed. However, the relationships between batch processing tasks in production are complex, making it difficult to determine the synchronization timing. The technical solution provided in this application is an optimization scheme for the timing of data synchronization initiation. Based on the above issues, to ensure timely synchronization, synchronization cannot be delayed until the core batch processing task is fully completed. However, synchronizing upon the completion of a single task may lead to data inconsistencies. Therefore, a reverse dependency detection method is used to achieve precise triggering of synchronization.

[0061] In this embodiment, the data table is processed after the completion information of any batch processing script is detected. Alternatively, other methods can be used to detect the data table. When preset conditions are met, it is determined whether the target task relationship pointing to the currently detected data table is complete. For example, the data table can be detected at preset intervals, which can be adjusted in real time according to business pressure. Compared to the solution used in this application, the method of detecting at preset intervals consumes fewer computing resources, but its real-time performance is far weaker than the technical solution provided in this application.

[0062] S12: Determine whether all target tasks have been completed;

[0063] When the current data cluster detects that any batch processing script task has finished running normally, it marks the completion of the task for that day in the task table dependency graph; it uses spark-catalyst to parse out the target table of this script; it uses the target table as a node and queries all one-way task relationships pointing to the target table in the graph (reverse detection relationship); it traverses all task relationships of the target table to confirm whether they are all completed; if not, it does not trigger the data synchronization task and continues to wait for the completion of the next batch processing task; if it has been completed, it starts the target table data synchronization task to synchronize the data to other target clusters that need the data, thus completing this synchronization task.

[0064] S13: If all is completed, synchronize the data in the target data table to each data warehouse.

[0065] It is understood that when executing the data synchronization method provided in this embodiment, it is also necessary to obtain and parse each batch processing script to obtain the batch processing task, source data table and target data table corresponding to each batch processing script; the source data table and target data table are used as nodes, and the batch processing tasks between the source data table and target data table are used as unidirectional task relationships to generate a task table dependency graph.

[0066] In practice, the source data table and the target data table have a many-to-one relationship, meaning that multiple source data tables can point to the same target data table to achieve data processing.

[0067] This embodiment provides a data synchronization method, including: upon detecting the completion information of any batch processing script, obtaining the target data table corresponding to the completion information; determining the target task relationships pointing unidirectionally to the target data table based on the task table dependency graph; determining whether all target task relationships are completed; and if all are completed, synchronizing the data in the target data table to each data warehouse. Therefore, the technical solution provided in this application, by analyzing the task relationships of each target data table, performs data synchronization only when all task relationships are completed, without waiting for the progress of other data tables in the cluster where the target data table resides. Data synchronization can be achieved during batch processing, thereby reducing the time occupied by the data processing process and improving data analysis efficiency.

[0068] Figure 2 This is a flowchart illustrating the generation process of the task table dependency graph provided in an embodiment of this application. Figure 3 This is a schematic diagram of the task table dependency graph provided in the embodiments of this application, such as... Figure 2 or Figure 3 As shown, before the step following the detection of completion information for any batch script, the following is also included:

[0069] Obtain and parse each batch script to obtain the corresponding batch processing task, source data table, and target data table; use the source data table and target data table as nodes, and the batch processing tasks between the source data table and target data table as a unidirectional task relationship to generate a task table dependency graph.

[0070] In practice, the Spark SQL parsing tool Catalyst is used to parse the batch script into source and target tables. The tables are used as nodes, and the task names between the source and target tables are used as unidirectional relationship edges. These are stored in a graph database to generate a task table dependency graph.

[0071] All batch processing scripts are parsed, and tables are stored as nodes and tasks as edges in a graph database. When task 2 in the graph finishes, the target table can be identified as table D. Using a graph algorithm that detects unidirectional edges by nodes, it can be identified that table D has two preceding tasks: task 1 and task 2. If task 1 is also completed at this time, the data of table D is immediately synchronized to the cluster that needs to receive the data, and batch processing of task 4 begins simultaneously; otherwise, the process continues to wait.

[0072] like Figure 3 As shown, Tables A, B, and C are source tables for Table D, and Table D is a source table for Table E. Once Task 1 and Task 2 are completed, all task relationships pointing to Table D are completed, and the data in Table D can be synchronized to other data clusters.

[0073] The synchronization task uses separate YARN resources, with a total of 10 nodes, 160GB of memory and 80 cores (16GB of memory / 8 cores per machine). Due to the limited resource pool, it is impossible to start all tasks at the same time. Therefore, a method is needed to ensure that high-priority tasks (manually defined important tasks) are synchronized first, while also taking into account synchronization efficiency and making all tasks complete as quickly as possible. Figure 4 A flowchart of a data synchronization method provided in an embodiment of this application is shown below. Figure 4 As shown, synchronizing data from the target data table to each data warehouse includes:

[0074] S40: Generate a data synchronization task based on the target data table;

[0075] S41: Send the data synchronization task to the cache pool and calculate the priority of the data synchronization task;

[0076] S42: Execute data synchronization tasks in the cache pool according to priority.

[0077] The task buffer pool approach detects and triggers a migration task, but does not immediately execute the data synchronization task. Instead, it places the task to be executed into the buffer pool. The priority calculator uses a pre-selection algorithm to periodically retrieve all tasks from the buffer pool and recalculate the priority of the tasks according to the resource priority algorithm. The resource determiner selects the next task based on the remaining resources.

[0078] Figure 5 A flowchart of a priority calculation method provided in an embodiment of this application is shown below. Figure 5 As shown, the priorities for computational data synchronization tasks include:

[0079] S50: Obtain the initial priority of the synchronization task and the historical data corresponding to the synchronization task;

[0080] S51: Based on historical data, predict the estimated resource consumption and estimated execution time of the synchronization task;

[0081] S52: Determine the priority based on the initial priority, estimated resource consumption, and estimated execution time.

[0082] Figure 6 This is a schematic diagram of future idle resources provided in the embodiments of this application, such as... Figure 6As shown, if the detection is performed at time 0, there are currently three tasks: Task 1, Task 2, and Task 3. According to historical data, the maximum data synchronization resources that can be used in the next t are the total system resources minus the unfinished portions of Tasks 1, 2, and 3. The earliest time that a task can be placed into the next task is the time r that Task 1 will complete. The available resources in the next t are represented by the gray area.

[0083] Cache pool priority allocation method:

[0084] Assuming the priority classification group is p, the manually defined priority is p0, and the waiting time is t (minutes), the final priority classification group can be calculated using the following formula, where a smaller priority value indicates a higher priority:

[0085] p = p0 - (t / 30)

[0086] Where (t / 30) represents the largest integer not exceeding t / 30. This formula means that the priority classification group increases by one unit every 30 minutes. For example, if the initial priority classification group is 3, and after waiting 60 minutes, the final priority classification group will be 1.

[0087] Currently, a priority group may contain multiple tasks. The execution order is determined based on the combined historical memory and CPU consumption of each task; the higher the value, the earlier it is executed. Assuming the estimated average memory usage is M (in GB), the estimated average CPU core usage is N, and the execution time is T (in seconds), the final execution score for this priority group can be expressed as:

[0088] Score = (M*T / 2GB) + (N*T)

[0089] Where M*T / 2GB represents the estimated memory usage, and N*T represents the estimated CPU core usage; the sum of the two is the final score. Based on the above two steps, a comprehensive priority is determined, with the task with the highest priority (lowest numerical value) and highest score being executed first. When cluster synchronization resources are released and the resource determiner makes a judgment, a new synchronization task is initiated if resources are sufficient, ensuring maximum utilization of synchronization resources and efficient completion of data synchronization.

[0090] In a preferred embodiment, after determining the priority based on the initial priority, estimated resource consumption, and estimated execution time, the method further includes: obtaining the execution information of the currently executing batch script; determining the amount of idle computing resources within a threshold time based on the execution information and historical data; and correcting the priority based on the amount of idle computing resources, estimated resource consumption, and estimated execution time.

[0091] Furthermore, executing data synchronization tasks in the cache pool according to priority includes: determining the tasks to be executed based on priority; obtaining the estimated resource consumption of the tasks to be executed; determining whether the current amount of idle resources is greater than the estimated resource consumption; and if it is greater than the estimated resource consumption, then executing the tasks to be executed.

[0092] The priority is adjusted according to a pre-defined program that prioritizes tasks. The input parameters are the task list, task synchronization start time, task data size, task resource consumption, and the algorithm used. The output parameters are the priority order in which tasks are retrieved from the buffer pool and the estimated resource consumption of each task.

[0093] The resource arbiter will determine the available resources in the cluster. If the required memory is less than 130% of the available memory and the required CPU is less than 150% of the idle CPU, the task will be started.

[0094] Assuming the required memory is M, the available memory is Fm, the required CPU is C, and the idle CPU is Fc, then it can be expressed by the following formula:

[0095] If M≤Fm×1.3 and C≤Fc×1.5, then the call is made. The optimal parameters are obtained by adjusting and verifying after multiple attempts, where memory is less than 1.3 times the idle resources and CPU is less than 1.5 times. Figure 7 This is a diagram illustrating the execution of a synchronous task, such as... Figure 7 As shown, this task pool is not a familiar stack or queue; tasks are not first-in, first-out (FIFO) or last-in, first-out (LIFO). Instead, each time a task enters the pool, the priority calculator recalculates the priority of all tasks in the pool to determine the next task to be executed. Since the waiting tasks in the pool do not actually use physical resources such as memory, CPU, or network I / O, it ensures that the resources of the executing tasks are used efficiently and there is no resource contention, thus maximizing the efficiency of the entire data synchronization process. The resource determiner periodically obtains the idle resource status on YARN and continuously calculates the highest priority task [Synchronization Task 3]. When it finds that the estimated memory requirement of [Synchronization Task 3] is less than 130% of the free memory and the estimated CPU requirement is less than 150% of the idle CPU, [Synchronization Task 3] is submitted and waits 10 seconds before the next check.

[0096] In the above embodiments, the data synchronization method has been described in detail. This application also provides embodiments corresponding to the data synchronization device. It should be noted that this application describes the embodiments of the device from two perspectives: one is based on the functional modules, and the other is based on the hardware.

[0097] Figure 8 This is a structural diagram of a data synchronization device provided in an embodiment of this application, as shown below. Figure 8As shown, it includes:

[0098] The acquisition module 10 is used to acquire the target data table corresponding to the completion information when the completion information of any batch processing script is detected.

[0099] Module 11 is used to determine the target task relationship that points unidirectionally to the target data table based on the task table dependency graph;

[0100] Module 12 is used to determine whether all target task relationships have been completed.

[0101] Synchronization module 13 is used to synchronize the data in the target data table to each data warehouse if all tasks are completed.

[0102] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0103] This embodiment provides a data synchronization device, comprising: upon detecting completion information of any batch processing script, obtaining the target data table corresponding to the completion information; determining the target task relationships pointing unidirectionally to the target data table based on the task table dependency graph; determining whether all target task relationships are completed; and if all are completed, synchronizing the data in the target data table to each data warehouse. Therefore, the technical solution provided in this application, by analyzing the task relationships of each target data table, performs data synchronization only when all task relationships are completed, without waiting for the progress of other data tables in the cluster where the target data table resides. Data synchronization can be achieved during batch processing, thereby reducing the time occupied by the data processing process and improving data analysis efficiency.

[0104] Figure 9 A structural diagram of a data synchronization device provided in another embodiment of this application is shown below. Figure 9 As shown, the data synchronization device includes: a memory 20 for storing computer programs;

[0105] The processor 21 is used to implement the steps of the data synchronization method as described in the above embodiments when executing a computer program.

[0106] The data synchronization device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0107] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0108] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the data synchronization method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, batch scripts, task dependency graphs, etc.

[0109] In some embodiments, the data synchronization device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0110] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the data synchronization device and may include more or fewer components than illustrated.

[0111] The data synchronization device provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method:

[0112] Once the completion information of any batch script is detected, the target data table corresponding to the completion information is obtained.

[0113] Determine the target task relationship that points unidirectionally to the target data table based on the task table dependency graph;

[0114] Determine whether all target tasks have been completed;

[0115] If all is completed, the data in the target data table will be synchronized to each data warehouse.

[0116] This embodiment provides a data synchronization device, comprising: upon detecting completion information of any batch processing script, obtaining the target data table corresponding to the completion information; determining the target task relationships pointing unidirectionally to the target data table based on the task table dependency graph; determining whether all target task relationships are completed; and if all are completed, synchronizing the data in the target data table to each data warehouse. Therefore, the technical solution provided in this application, by analyzing the task relationships of each target data table, performs data synchronization only when all task relationships are completed, without waiting for the progress of other data tables in the cluster where the target data table resides. Data synchronization can be achieved during batch processing, thereby reducing the time occupied by the data processing process and improving data analysis efficiency.

[0117] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0118] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] The data synchronization method, apparatus, and medium provided in this application have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0120] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method of data synchronization, the method comprising: The method comprises the following steps: When detecting the completion information of any batch processing script, obtaining the target data table corresponding to the completion information; According to the task table dependency graph, determining the target task relationship unidirectionally pointing to the target data table; Judging whether all the target task relationships are completed; If all the target task relationships are completed, synchronizing the data in the target data table to each data warehouse; Before the step of detecting the completion information of any batch processing script, the method further comprises the following steps: Obtaining each batch processing script and parsing to obtain the batch processing task, the source data table and the target data table corresponding to each batch processing script; Taking the source data table and the target data table as nodes and the batch processing task between the source data table and the target data table as a unidirectional task relationship, to generate the task table dependency graph; The step of synchronizing the data in the target data table to each data warehouse comprises the following steps: Generating a data synchronization task according to the target data table; Sending the data synchronization task to a cache pool and calculating the priority of the data synchronization task; Executing the data synchronization task in the cache pool according to the priority; The step of calculating the priority of the data synchronization task comprises the following steps: Determining the priority classification group to which the data synchronization task belongs: p = p0 - (t / 30); In the formula, p is the priority classification group, p0 is the initial priority, t is the waiting time, and (t / 30) represents the maximum integer not exceeding t / 30; Determining the execution score value corresponding to each data synchronization task in the priority classification group: Execution score value = (M*T / 2GB) + (N*T); In the formula, M is the estimated average memory, N is the estimated average number of CPU cores used, and T is the estimated execution time; Based on the corresponding priority classification group and the execution score value, the priority of the data synchronization task is determined.

2. The data synchronization method of claim 1, wherein, The step of calculating the priority of the data synchronization task comprises the following steps: Obtaining the initial priority of the synchronization task and the historical data corresponding to the synchronization task; According to the historical data, predicting the estimated resource consumption and the estimated execution time of the synchronization task; According to the initial priority, the estimated resource consumption and the estimated execution time, the priority is determined.

3. The data synchronization method of claim 2, wherein, After the step of determining the priority according to the initial priority, the estimated resource consumption and the estimated execution time, the method further comprises the following steps: Obtaining the execution information of the currently executed batch processing script; According to the execution information and the historical data, determining the amount of idle computing resources within the threshold time; According to the amount of idle computing resources and the estimated resource consumption and the estimated execution time, the priority is corrected.

4. The data synchronization method of claim 1, wherein, The step of executing the data synchronization task in the cache pool according to the priority comprises the following steps: According to the priority, determining the to-be-executed task; Obtaining the estimated resource consumption of the to-be-executed task; Judging whether the current amount of idle resources is greater than the estimated resource consumption; If the current amount of idle resources is greater than the estimated resource consumption, the to-be-executed task is executed.

5. The data synchronization method of claim 1, wherein, The step of obtaining the target data table corresponding to the completion information comprises the following steps: The batch processing script corresponding to the completion information is parsed by using a spark-SQL parsing tool to obtain the target data table.

6. A data synchronization apparatus, characterized by comprising: The method comprises the steps of: acquiring a target data table corresponding to completion information of any batch processing script when the completion information is detected; determining a target task relationship of the target data table in one direction according to a task table dependency graph; judging whether all the target task relationships are completed; synchronizing data in the target data table to each data warehouse if all the target task relationships are completed; wherein the data synchronization device is further configured to: acquire each batch processing script and parse the batch processing script to obtain a batch processing task, a source data table and the target data table corresponding to each batch processing script; take the source data table and the target data table as nodes and the batch processing task between the source data table and the target data table as a one-way task relationship to generate the task table dependency graph; the synchronization module is configured to generate a data synchronization task according to the target data table, send the data synchronization task to a cache pool and calculate a priority of the data synchronization task, and execute the data synchronization task in the cache pool according to the priority; the synchronization module is configured to: determine a priority classification group to which the data synchronization task belongs: p = p0 - (t / 30); wherein p is the priority classification group, p0 is an initial priority, t is a waiting time, and (t / 30) represents a maximum integer not exceeding t / 30; determine an execution score value corresponding to each data synchronization task in the priority classification group: execution score value = (M*T / 2GB) + (N*T); wherein M is an estimated average memory, N is an estimated average number of CPU cores, and T is an estimated execution time; and determine the priority of the data synchronization task based on the corresponding priority classification group and the execution score value.

7. A data synchronization apparatus, characterized by comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the data synchronization method according to any one of claims 1 to 5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the data synchronization method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the data synchronization method according to any one of claims 1 to 5.

Images