Method and device for synchronizing data in the order of billions based on distributed environment and storage medium

Abstract

The application relates to an order of magnitude of 100 million data synchronization method and device based on a distributed environment and a storage medium, applied to the technical field of big data synchronization, and comprising the following steps: taking a distributed computer as the basis, splitting a data synchronization task, making multiple computers simultaneously synchronize the split subtasks, greatly improving the data synchronization rate, and deploying the distributed computer so that the user can freely adjust the number of computers needed to work according to the amount of data needed to be synchronized; and in the data synchronization process, a specified sharding algorithm in a scheduling method is used to operate with the number of target tables in a target data source, so that it is determined that each piece of data in an original data source should be synchronized to which target table in the target data source, the single-table data in the original data source is split into multi-table data in the target data source, and the read-write pressure of each target table in the target data source can be effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of big data synchronization technology, specifically to a method, apparatus, and storage medium for synchronizing hundreds of millions of data points in a distributed environment. Background Technology

[0002] In existing big data synchronization technologies, a single computer is typically used to complete data synchronization. Under the background requirement of synchronizing hundreds of millions of data points, the synchronization rate is very slow. Moreover, the fixed use of a single computer offers no scalability and cannot achieve the goal of flexibly increasing or decreasing the number of computers according to the size of the data. At the same time, existing technologies do not support splitting single table data in the original data source into multiple data tables in the target data source during the data synchronization process, resulting in a large read and write pressure on a single target table. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, apparatus and storage medium for synchronizing hundreds of millions of data points in a distributed environment, so as to solve the problems in the prior art where data synchronization is completed by a single computer, the synchronization rate is slow when the data to be synchronized is large, and the number of computers cannot be flexibly increased or decreased according to the data volume. At the same time, it solves the problem that the existing data synchronization process does not support splitting single table data in the original data source into multiple data tables in the target data source, resulting in a large read and write pressure on a single target table.

[0004] According to a first aspect of the present invention, a method for synchronizing hundreds of millions of data points in a distributed environment is provided, the method comprising:

[0005] Configure the scheduling method for distributed computers in the distributed task scheduling tool, and trigger the deployed distributed computers through the distributed task scheduling tool;

[0006] Once the distributed computer is triggered, the management computer and multiple sub-task computers are determined according to the scheduling method of the distributed computer configured in the distributed task scheduling tool.

[0007] The management computer divides the data synchronization task into multiple subtasks according to the scheduling method, and assigns the divided subtasks to itself and multiple subtask computers.

[0008] The management computer and subtask computers read data from the original data source according to their own subtasks, and perform calculations according to the sharding algorithm and the number of target tables specified in the scheduling method, so as to determine which target table in the target data source each piece of data in the original data source should be synchronized to, until the goal of synchronizing all original data to the target data source is achieved.

[0009] Priority,

[0010] The management computer divides the data synchronization task into multiple subtasks according to a scheduling method, and assigns the divided subtasks to itself and multiple subtask computers, including:

[0011] After the management computer splits the data synchronization task into multiple subtasks according to the scheduling method, it persists the split subtasks to the task manager. Before running data synchronization, all subtask computers and the management computer continuously query the task manager in a timed polling manner until the task is obtained and then the polling stops.

[0012] Preferably,

[0013] After the management computer or subtask computer obtains the subtask, it further splits the subtask according to the scheduling method to obtain the second subtask. The management computer or subtask computer then synchronously executes the second subtask split from its own subtask.

[0014] Preferably,

[0015] The step of calculating, based on the sharding algorithm specified in the scheduling method and the number of target tables, to determine which target table in the target data source each piece of data in the original data source should be synchronized to includes:

[0016] Sort all data tables in the target data source, and each data table has its corresponding sequence number;

[0017] The management computer or subtask computer, based on its second subtask, parses the value of each column of the corresponding data from the original data source as a shard value, calculates the remainder between the shard value and the number of data tables in the target data source, and this remainder is the sequence number of the data column in the original data source that should be stored in the data table of the target data source.

[0018] Preferably,

[0019] The scheduling method also includes a data synchronization mode;

[0020] The data synchronization modes include full synchronization mode and full plus incremental synchronization mode.

[0021] The full synchronization mode includes:

[0022] When the management computer or subtask computer starts executing a subtask, all data in the original data source at the time when the subtask starts executing is synchronized to the target data source.

[0023] The full-plus-incremental synchronization mode includes:

[0024] The database plugin detects incremental data generated during the full synchronization process and collects the incremental data into a distributed queue.

[0025] After full synchronization is complete, all management computers or subtask computers that have completed full synchronization will automatically consume the incremental data from the distributed queue.

[0026] After consuming incremental data, the incremental data is still split into multiple sub-tasks. The management computer or the sub-task computer synchronizes the incremental data to the target table of the target data source according to the obtained sub-tasks.

[0027] Preferably,

[0028] After the management computer breaks down the data synchronization task into multiple subtasks, it synchronizes the results of the subtasks to the status manager. Each time the management computer and the subtask computer complete their own second subtask, the management computer synchronizes the completed status to the status manager.

[0029] Preferably,

[0030] After the management computer or subtask computer obtains the raw data from the original data source, it directly packages the raw data and sends it to the user as a method parameter.

[0031] Users perform various processing operations on the raw data in the form of method parameters, and send the processed data to the management computer or sub-task computer, which then synchronizes the processed data to the target data source.

[0032] According to a second aspect of the present invention, a device for synchronizing hundreds of millions of data points in a distributed environment is provided, the device comprising:

[0033] Task triggering module: Used to configure the scheduling method of distributed computers in the distributed task scheduling tool, and trigger the deployed distributed computers through the distributed task scheduling tool;

[0034] Computer classification module: After a distributed computer is triggered, it determines the management computer and multiple sub-task computers according to the scheduling method of the distributed computer configured in the distributed task scheduling tool.

[0035] Task splitting module: used by the management computer to split the data synchronization task into multiple subtasks according to the scheduling method, and to allocate the split subtasks to itself and multiple subtask computers;

[0036] Data synchronization module: The management computer and subtask computers read data from the original data source according to their own subtasks, and perform calculations according to the sharding algorithm and the number of target tables specified in the scheduling method, so as to determine which target table in the target data source each piece of data in the original data source should be synchronized to, until the goal of synchronizing all original data to the target data source is achieved.

[0037] According to a third aspect of the present invention, a storage medium is provided, the storage medium storing a computer program, which, when executed by a host controller, implements the steps of the above-described method.

[0038] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0039] This application uses a distributed computing approach to break down data synchronization tasks, allowing multiple computers to simultaneously synchronize the sub-tasks. This significantly improves the data synchronization rate. Furthermore, the distributed computing deployment allows users to freely adjust the number of computers required to synchronize the amount of data. During data synchronization, the sharding algorithm specified in the scheduling method is used to calculate the number of target tables in the target data source, determining which target table in the target data source each piece of data in the original data source should be synchronized to. This effectively splits single-table data in the original data source into multi-table data in the target data source, thus reducing the read and write pressure on each target table in the target data source.

[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0042] Figure 1 This is a flowchart illustrating a method for synchronizing hundreds of millions of data points in a distributed environment, according to an exemplary embodiment.

[0043] Figure 2 This is a schematic diagram illustrating the overall principle of a method for synchronizing hundreds of millions of data points in a distributed environment, based on an exemplary embodiment.

[0044] Figure 3 This is a schematic diagram illustrating the principle of distributed data parallel synchronization according to an exemplary embodiment;

[0045] Figure 4 This is a schematic diagram illustrating the principle of task sharding within a single computer according to an exemplary embodiment;

[0046] Figure 5 This is a schematic diagram illustrating the principle of table-splitting data synchronization according to an exemplary embodiment;

[0047] Figure 6 This is a schematic diagram illustrating the principle of full-plus-increment synchronization mode according to an exemplary embodiment;

[0048] Figure 7 This is a schematic diagram of a system for synchronizing hundreds of millions of data points in a distributed environment, according to an exemplary embodiment.

[0049] In the attached diagram: 1-Task triggering module, 2-Computer classification module, 3-Task splitting module, 4-Data synchronization module. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0051] Example 1

[0052] Figure 1 This is a flowchart illustrating a method for synchronizing hundreds of millions of data points in a distributed environment, based on an exemplary embodiment. Figure 1 As shown, the method includes:

[0053] S1. Configure the scheduling method of the distributed computers in the distributed task scheduling tool, and trigger the deployed distributed computers through the distributed task scheduling tool;

[0054] S2, after the distributed computer is triggered, the management computer and multiple sub-task computers are determined according to the scheduling method of the distributed computer configured in the distributed task scheduling tool;

[0055] S3, the management computer divides the data synchronization task into multiple subtasks according to the scheduling method, and assigns the divided subtasks to itself and multiple subtask computers;

[0056] S4, the management computer and the subtask computer read data from the original data source according to their own subtasks, and perform calculations according to the sharding algorithm and the number of target tables specified in the scheduling method, so as to determine which target table of the target data source each piece of data in the original data source should be synchronized to, until the goal of synchronizing all original data to the target data source is achieved.

[0057] Understandably, as shown in the attached document Figure 2 As shown, this application is based on distributed computing. Users write data synchronization jobs in JSON data format and import them into a distributed scheduling tool. The distributed scheduling tool triggers pre-deployed distributed environment computers based on the synchronization jobs written by the user. Each computer, upon receiving the trigger from the scheduling tool, performs the three-step data synchronization process: data extraction, data processing, and data loading, ultimately achieving data synchronization. Specifically, after the distributed computers are triggered, the management computer (master) and multiple sub-task computers (workers) are determined according to the scheduling method configured in the distributed task scheduling tool, as shown in the attached diagram. Figure 3 As shown, the management computer divides the data synchronization task into multiple subtasks according to the scheduling method, and assigns the divided subtasks to itself and multiple subtask computers. The management computer and the subtask computers read data from the original data source according to their own subtasks, and perform calculations according to the sharding algorithm specified in the scheduling method and the number of target tables, so as to determine which target table of the target data source each piece of data in the original data source should be synchronized to, until the goal of synchronizing all original data to the target data source is achieved.

[0058] It is worth noting that the following is a data synchronization configuration for the scheduling tool submitted by a user:

[0059] {

[0060] "threadNum": 10, / / Parallelism capability of a single computer

[0061] "leaderIndex":0, / / The number of the computer in the cluster that assumes the Master role

[0062] "taskItemSize": 10, / / Parallel processing capability of the entire task

[0063] "limitStepSize": 5000, / / JDBC polling, the amount of raw data retrieved each time.

[0064] "etlMode":"FULL_UPDATE", / / Data synchronization mode

[0065] "startDateTime":"2020-01-01 00:00:00", / / Original data start time

[0066] "endDateTime":"2022-03-31 19:01:00", / / End time of the original data

[0067] "tableName":"original_table_name", / / Original table name

[0068] "shardingColumnIndex":1, / / The sharding key's index in the original record

[0069] "targetTablePrefix":"target_table_prefix", / / Target table prefix

[0070] "targetTableNums":20, / / Number of target tables

[0071] "shardingStrategy":"HASH" / / Sharding algorithm

[0072] }

[0073] Priority,

[0074] The management computer divides the data synchronization task into multiple subtasks according to a scheduling method, and assigns the divided subtasks to itself and multiple subtask computers, including:

[0075] After the management computer splits the data synchronization task into multiple sub-tasks according to the scheduling method, it persists the split sub-tasks to the task manager. Before running data synchronization, all sub-task computers and the management computer continuously query the task manager in a timed polling manner until the task is obtained and then the polling stops.

[0076] Understandably, as shown in the attached document Figure 3As shown, the core of this invention's support for distributed data synchronization lies in the computer cluster deployment and task splitting mechanism. Firstly, a crucial prerequisite is that the user configures multiple computers to form a cluster for data synchronization. Secondly, the computers in the cluster are divided into two roles: Master and Worker. The most basic function of both roles is to run the synchronization tasks they acquire. In addition, the Master role is responsible for overall data synchronization coordination. Before all computers synchronize, it divides the tasks according to the number of computers and the total amount of data to be synchronized. For example, if the original data source has 100 million records to be synchronized, for the above synchronization configuration, the program will read the `taskItemSize` field value. Here, the user-provided configuration is 10. Therefore, the 100 million records will be split into 10 synchronization tasks by the Master, each task responsible for synchronizing 10 million records. The Master persists the split tasks to the task manager. Before running data synchronization, all computers continuously query the task manager in a periodic polling manner until a task is obtained, at which point the polling stops, and finally, data synchronization begins. In this way, by using the divide-and-conquer approach, large tasks are broken down into smaller tasks, which are then run in parallel by the cluster, thus greatly improving the synchronization rate.

[0077] Preferably,

[0078] After the management computer or subtask computer obtains the subtask, it further splits the subtask according to the scheduling method to obtain the second subtask. The management computer or subtask computer synchronously executes the second subtask split from its own subtask.

[0079] Understandably, as shown in the attached document Figure 4 As shown, after each subtask computer obtains its own subtask, in order to further accelerate the timeliness of data synchronization and improve synchronization efficiency, each computer will further split the task it obtained into multiple executable shard tasks according to the data sharding strategy configured by the user. After the subtask computer or management computer in the cluster obtains a task to synchronize 10 million data points from the task manager, it will read the threadNum field value configured by the user. Here, the user-provided configuration is 10. Then, the computer will start 10 worker threads to synchronize the 10 million data points to be synchronized in parallel. Each worker thread is responsible for synchronizing 1 million data points, further improving the synchronization rate.

[0080] Preferably,

[0081] The step of calculating, based on the sharding algorithm specified in the scheduling method and the number of target tables, to determine which target table in the target data source each piece of data in the original data source should be synchronized to includes:

[0082] Sort all data tables in the target data source, and each data table has its corresponding sequence number;

[0083] The management computer or subtask computer, based on its own second subtask, parses the value of each column of the corresponding data from the original data source as a shard value, calculates the remainder between the shard value and the number of data tables in the target data source, and the remainder is the sequence number of the data column in the original data source that should be stored in the data table in the target data source.

[0084] Understandably, the significance of this invention lies in reducing the read and write pressure on the data source. For example, for a table with an original data volume of 100 million records, the data volume far exceeds the capacity of a traditional database, inevitably impacting business operations. In this context, this invention supports splitting the original data into multiple target tables to alleviate the read and write pressure on the new data source, specifically as shown in the attached table. Figure 5As shown, the process involves taking a piece of raw data and performing an algorithmic calculation based on the specified sharding algorithm (Hash, Mod) and the number of target tables to determine which target table the raw data should be synchronized to. This data is then written to the corresponding data buffer. When a target table accumulates a certain amount of data, a batch loading operation is performed. For this data synchronization configuration, the subtask computer or management computer reads the `shardingColumnIndex`, `targetTablePrefix`, `targetTableNums`, and `shardingStrategy` configurations. `shardingColumnIndex` is a pointer to the column number of the original table used as the sharding reference; `targetTablePrefix` is the prefix of the target table; `targetTableNums` is the number of target tables; and `shardingStrategy` is the data sharding algorithm. In this section, `shardingColumnIndex` is 1, `targetTablePrefix` is `target_table_prefix`, `targetTableNums` is 20, and `shardingStrategy` is H. ASH, then target_table_prefix[Hash(original data, first column value)%20], means that after obtaining the value of the first column of the original data, if it is 20, and there are 20 target tables in the target data source, the remainder of 20 is 0. The 20 target tables are sorted from 0 to 19. This means the data in the first column should be placed in the 0th table of the target data source. If the value of the Xth column is 28, then the remainder of 28 and 20 is 8, so the data in the Xth column is placed in the 8th target table. The remainder represents the number of complete values ​​contained in the shard value. Besides 20, the remaining values ​​mean that if the shard value is 20, then it contains a complete 20 and has a remainder of 0, which is the 0th target table. If the shard value is 28, then it contains a complete 20 and has a remainder of 8. If the shard value is 39, then it contains one complete 20 and has a remainder of 19, so this column of data should be placed in the 19th target table of the target data source. If the shard value is 47, then it contains two complete 20s and has a remainder of 7, so it should be placed in the 7th target table. In this way, the single-table data in the original data source is transformed into multi-table data in the target data source, thereby reducing the read and write pressure of a single table in the target table.

[0085] Preferably,

[0086] The scheduling method also includes a data synchronization mode;

[0087] The data synchronization modes include full synchronization mode and full plus incremental synchronization mode.

[0088] The full synchronization mode includes:

[0089] When the management computer or subtask computer starts executing a subtask, all data in the original data source at the time when the subtask starts executing is synchronized to the target data source.

[0090] The full-plus-incremental synchronization mode includes:

[0091] The database plugin detects incremental data generated during the full synchronization process and collects the incremental data into a distributed queue.

[0092] After full synchronization is complete, all management computers or subtask computers that have completed full synchronization will automatically consume the incremental data from the distributed queue.

[0093] After consuming incremental data, the incremental data is still split into multiple sub-tasks. The management computer or sub-task computer synchronizes the data to the target table of the target data source according to the obtained sub-tasks.

[0094] Understandably, as shown in the attached document Figure 6 As shown, the full synchronization mode is suitable for use when the original data source is broken. In this mode, the computer only needs to continuously extract data through the JDBC interface, process the data, and shard the data before finally loading it into the target data source. In other words, in the full synchronization mode, the management computer and the sub-task computer only need to process the data in the original data source from the time node where the task starts. If the data in the original data source changes during the synchronization process, the changed data will not be synchronized. In the full plus incremental synchronization mode, when the data in the original data source changes during the data synchronization process, the changed data will also be synchronized until the data in the original data source no longer changes during a certain data synchronization process. The check mode is convenient for checking the data at both ends after the full data synchronization is completed. If inconsistent data is found at both ends, the data at both ends will be recorded in the log table for easy manual intervention and adjustment. For the synchronization configuration in this chapter, the system will read the etlMode field value: FULL_UPDATE, indicating that the synchronization task uses the full plus incremental processing mode.

[0095] Preferably,

[0096] After the management computer breaks down the data synchronization task into multiple sub-tasks, it synchronizes the results of the sub-tasks to the status manager. After each of the management computer and the sub-task computers completes its own second sub-task, the management computer synchronizes the completed status to the status manager.

[0097] It is understandable that this invention involves two splits of the original task. The first is the initial split by the Master role, and the second is the secondary splitting operation performed by the computer on the received task. After each split, the results are persistently saved to the state manager. In addition, each computer also persists the amount of data that it has successfully synchronized into the state manager after synchronization is completed, which is convenient for later traceability.

[0098] Preferably,

[0099] After the management computer or subtask computer obtains the raw data from the original data source, it directly packages the raw data and sends it to the user as a method parameter.

[0100] Users perform various processing operations on the raw data in the form of method parameters, and send the processed data to the management computer or sub-task computer, which then synchronizes the processed data to the target data source.

[0101] It is understandable that by exposing interfaces, users can customize and write data processing logic, such as sending HTTP requests, converting raw data formats, performing logical operations, etc. The principle is to directly package the raw data and pass it to the user as a method parameter so that the user can perform various processing on the raw data, rather than increasing the complexity of data processing through various forms of encapsulation or only supporting users to perform simple data processing.

[0102] Example 2

[0103] Figure 7 This is a schematic diagram of a system for synchronizing hundreds of millions of data points in a distributed environment, according to an exemplary embodiment, including:

[0104] Task Trigger Module 1: Used to configure the scheduling method of distributed computers in the distributed task scheduling tool, and trigger the deployed distributed computers through the distributed task scheduling tool;

[0105] Computer Classification Module 2: After a distributed computer is triggered, it determines the management computer and multiple sub-task computers according to the scheduling method of the distributed computer configured in the distributed task scheduling tool.

[0106] Task splitting module 3: used by the management computer to split the data synchronization task into multiple subtasks according to the scheduling method, and to allocate the split subtasks to itself and multiple subtask computers;

[0107] Data synchronization module 4: The management computer and subtask computer read data from the original data source according to their own subtasks, and perform calculations according to the sharding algorithm and the number of target tables specified in the scheduling method, so as to determine which target table of the target data source each piece of data in the original data source should be synchronized to, until the goal of synchronizing all original data to the target data source is achieved.

[0108] Understandably, in this embodiment, the task triggering module 1 is used to configure the scheduling method of the distributed computers in the distributed task scheduling tool, and the deployed distributed computers are triggered by the distributed task scheduling tool; the computer classification module 2 is used to determine the management computer and multiple sub-task computers according to the scheduling method of the distributed computers configured in the distributed task scheduling tool after the distributed computers are triggered; the task splitting module 3 is used by the management computer to split the data synchronization task into multiple sub-tasks according to the scheduling method, and to allocate the split sub-tasks to itself and multiple sub-task computers; the data synchronization module 4 is used by the management computer and the sub-task computers to read data from the original data source according to their own sub-tasks, and to perform calculations according to the sharding algorithm and the number of target tables specified in the scheduling method, thereby obtaining the original data. This application determines which target table in the target data source each piece of data should be synchronized to until all original data is synchronized to the target data source. It utilizes a distributed computing approach, breaking down the data synchronization task into smaller parts, allowing multiple computers to simultaneously synchronize the sub-tasks. This significantly improves the data synchronization rate. Furthermore, the distributed computing deployment allows users to freely adjust the number of computers needed to synchronize the required amount of data. During data synchronization, the sharding algorithm specified in the scheduling method is used to calculate the number of target tables in the target data source, determining which target table in the target data source each piece of data should be synchronized to. This effectively splits single-table data in the original data source into multiple-table data in the target data source, significantly reducing the read / write pressure on each target table.

[0109] Example 3:

[0110] This embodiment provides a storage medium storing a computer program, which, when executed by a host controller, implements the various steps in the above method.

[0111] It is understood that the storage medium mentioned above can be a read-only memory, a hard disk, or an optical disk, etc.

[0112] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0113] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0114] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0115] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0117] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0118] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0119] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0120] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for synchronizing hundreds of millions of data points in a distributed environment, characterized in that: The method includes: Configure the scheduling method for distributed computers in the distributed task scheduling tool, and trigger the deployed distributed computers through the distributed task scheduling tool; Once the distributed computer is triggered, the management computer and multiple sub-task computers are determined according to the scheduling method of the distributed computer configured in the distributed task scheduling tool. The management computer divides the data synchronization task into multiple subtasks according to the scheduling method, and assigns the divided subtasks to itself and multiple subtask computers. The management computer divides the data synchronization task into multiple subtasks according to a scheduling method, and assigns the divided subtasks to itself and multiple subtask computers, including: After the management computer splits the data synchronization task into multiple sub-tasks according to the scheduling method, it persists the split sub-tasks to the task manager. Before running data synchronization, all sub-task computers and the management computer continuously query the task manager in a timed polling manner until the task is obtained and then the polling stops. The management computer and subtask computers read data from the original data source according to their own subtasks, and perform calculations according to the sharding algorithm and the number of target tables specified in the scheduling method, so as to determine which target table in the target data source each piece of data in the original data source should be synchronized to, until the goal of synchronizing all original data to the target data source is achieved. After the management computer or subtask computer obtains the subtask, it further splits the subtask according to the scheduling method to obtain the second subtask. The management computer or subtask computer synchronously executes the second subtask split from its own subtask. The step of calculating, based on the sharding algorithm specified in the scheduling method and the number of target tables, to determine which target table in the target data source each piece of data in the original data source should be synchronized to includes: Sort all data tables in the target data source, and each data table has its corresponding sequence number; The management computer or subtask computer, based on its second subtask, parses the value of each column of the corresponding data from the original data source as a shard value, calculates the remainder between the shard value and the number of data tables in the target data source, and this remainder is the sequence number of the data column in the original data source that should be stored in the data table of the target data source.

2. The method according to claim 1, characterized in that, The scheduling method also includes a data synchronization mode; The data synchronization modes include full synchronization mode and full plus incremental synchronization mode. The full synchronization mode includes: When the management computer or subtask computer starts executing a subtask, all data in the original data source at the time when the subtask starts executing is synchronized to the target data source. The full-plus-incremental synchronization mode includes: The database plugin detects incremental data generated during the full synchronization process and collects the incremental data into a distributed queue. After full synchronization is complete, all management computers or subtask computers that have completed full synchronization will automatically consume the incremental data from the distributed queue. After consuming incremental data, the incremental data is still split into multiple sub-tasks. The management computer or the sub-task computer synchronizes the incremental data to the target table of the target data source according to the obtained sub-tasks.

3. The method according to claim 2, characterized in that, After the management computer breaks down the data synchronization task into multiple subtasks, it synchronizes the results of the subtasks to the status manager. Each time the management computer and the subtask computers complete their own second subtask, the management computer synchronizes the completed status to the status manager.

4. The method according to claim 1, characterized in that, After the management computer or subtask computer obtains the raw data from the original data source, it directly packages the raw data and sends it to the user as a method parameter. Users perform various processing operations on the raw data in the form of method parameters, and send the processed data to the management computer or sub-task computer, which then synchronizes the processed data to the target data source.

5. A data synchronization device for hundreds of millions of data points in a distributed environment, characterized in that: The device includes: Task triggering module: Used to configure the scheduling method of distributed computers in the distributed task scheduling tool, and trigger the deployed distributed computers through the distributed task scheduling tool; Computer classification module: After a distributed computer is triggered, it determines the management computer and multiple sub-task computers according to the scheduling method of the distributed computer configured in the distributed task scheduling tool. The management computer divides the data synchronization task into multiple subtasks according to a scheduling method, and assigns the divided subtasks to itself and multiple subtask computers, including: After the management computer splits the data synchronization task into multiple sub-tasks according to the scheduling method, it persists the split sub-tasks to the task manager. Before running data synchronization, all sub-task computers and the management computer continuously query the task manager in a timed polling manner until the task is obtained and then the polling stops. Task splitting module: used by the management computer to split the data synchronization task into multiple subtasks according to the scheduling method, and to allocate the split subtasks to itself and multiple subtask computers; Data synchronization module: The management computer and subtask computers read data from the original data source according to their own subtasks, and perform calculations according to the sharding algorithm and the number of target tables specified in the scheduling method, so as to determine which target table of the target data source each piece of data in the original data source should be synchronized to, until the goal of synchronizing all original data to the target data source is achieved. After the management computer or subtask computer obtains the subtask, it further splits the subtask according to the scheduling method to obtain the second subtask. The management computer or subtask computer synchronously executes the second subtask split from its own subtask. The step of calculating, based on the sharding algorithm specified in the scheduling method and the number of target tables, to determine which target table in the target data source each piece of data in the original data source should be synchronized to includes: Sort all data tables in the target data source, and each data table has its corresponding sequence number; The management computer or subtask computer, based on its second subtask, parses the value of each column of the corresponding data from the original data source as a shard value, calculates the remainder between the shard value and the number of data tables in the target data source, and this remainder is the sequence number of the data column in the original data source that should be stored in the data table of the target data source.

6. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by the host, implements each step of the method for synchronizing billions of data points in a distributed environment as described in any one of claims 1-4.

Images