Data synchronization method, device, computer equipment, and storage medium

Abstract

The present application relates to a data synchronization method, apparatus, computer device, storage medium, and computer program product. The method includes: obtaining the current number of synchronization tasks in a first database; when the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database based on the current number of synchronization tasks in the first database; synchronizing incremental data corresponding to the first number of synchronization tasks to the second database, and deleting the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold; and synchronizing the incremental data corresponding to the synchronization tasks from the first database and the second database to a third database in parallel. This method can reduce data latency in synchronization tasks.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] TiCDC (TiDB Change Data Capture) is a synchronization tool for synchronizing TiDB database data with Kafka or MySQL databases. It implements fast synchronization tasks through library-level TiCDC and table-level TiCDC.

[0003] When the number of synchronization tasks exceeds a certain value, for example, when the number of synchronization tasks exceeds 700, the synchronization efficiency will decrease and the latency will be high when using TiCDC for synchronization. If the synchronization latency is too high, downstream Kafka or MySQL databases will not be able to obtain the latest data in a timely manner, which will cause some problems, such as data display delays and inaccurate data calculation and comparison. Therefore, low latency of data synchronization is an important indicator of synchronization tasks. Summary of the Invention

[0004] Based on this, it is necessary to provide a data synchronization method, apparatus, computer equipment, computer-readable storage medium and computer program product that can reduce delays in order to address the above technical problems.

[0005] In a first aspect, the present application provides a data synchronization method. The method includes:

[0006] Get the current number of synchronization tasks in the first database;

[0007] When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database;

[0008] Synchronize incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0009] The incremental data corresponding to the synchronization task is synchronized from the first database and the second database to the third database in parallel.

[0010] In one embodiment, when there are multiple second databases, the incremental data corresponding to the first number of synchronization tasks are synchronized to the second databases, including: obtaining the number of databases of the second databases, and evenly distributing the first number of synchronization tasks to the number of second databases; synchronizing the incremental data corresponding to the synchronization tasks from the first database to the corresponding second databases.

[0011] In one embodiment, after synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database, the method also includes: updating the number of synchronization tasks in the first database; when the number of synchronization tasks in the updated first database is greater than the first threshold, creating a new fourth database; synchronizing the incremental data corresponding to the second number of synchronization tasks to the fourth database, and deleting the second number of synchronization tasks that have been synchronized in the first database, wherein the difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold; and synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fourth database to the third database in parallel.

[0012] In one embodiment, the data synchronization method also includes: obtaining the number of synchronization tasks in the second database; creating a fifth database when the number of synchronization tasks in the second database is greater than a first threshold; synchronizing the incremental data corresponding to the third number of synchronization tasks from the first database to the fifth database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0013] In one embodiment, when the number of synchronization tasks in the second database is greater than a first threshold, after creating the fifth database, the method further includes: synchronizing the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and deleting the synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; and synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0014] In one embodiment, before synchronizing the incremental data corresponding to the synchronization task to the second database, it also includes: obtaining the full data corresponding to the synchronization task in the first database; and synchronizing the full data corresponding to the synchronization task in the first database to the second database.

[0015] In a second aspect, the present application also provides a data synchronization device. The device includes:

[0016] A task quantity determination module, configured to obtain the current quantity of synchronization tasks in the first database;

[0017] A database creation module, configured to create a corresponding number of second databases according to the current number of synchronization tasks in the first database when the current number of synchronization tasks in the first database is greater than a first threshold;

[0018] A first synchronization module is configured to synchronize incremental data corresponding to a first number of synchronization tasks to a second database, delete the first number of synchronization tasks in the first database, obtain a remaining number of synchronization tasks in the first database, and make the remaining number of synchronization tasks in the first database less than or equal to a first threshold;

[0019] The second synchronization module is used to synchronize the incremental data corresponding to the synchronization task from the first database and the second database to the third database in parallel.

[0020] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are performed:

[0021] Get the current number of synchronization tasks in the first database;

[0022] When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database;

[0023] Synchronize incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0024] The incremental data corresponding to the synchronization task is synchronized from the first database and the second database to the third database in parallel.

[0025] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0026] Get the current number of synchronization tasks in the first database;

[0027] When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database;

[0028] Synchronize incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0029] The incremental data corresponding to the synchronization task is synchronized from the first database and the second database to the third database in parallel.

[0030] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0031] Get the current number of synchronization tasks in the first database;

[0032] When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database;

[0033] Synchronize incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0034] The incremental data corresponding to the synchronization task is synchronized from the first database and the second database to the third database in parallel.

[0035] The above-mentioned data synchronization method, apparatus, computer equipment, storage medium and computer program product obtain the current number of synchronization tasks in the first database. When the current number of synchronization tasks in the first database is greater than a first threshold, a second database is created based on the current number of synchronization tasks in the first database, and the incremental data corresponding to the first number of synchronization tasks is synchronized to the second database. The first number of synchronization tasks in the first database is deleted, wherein the difference between the current number and the first number is less than or equal to the first threshold, and the incremental data corresponding to the synchronization tasks is synchronized from the first database and the second database to the third database. The above-mentioned data synchronization method, when the current number of synchronization tasks in the first database is greater than the first threshold, creates a new second database, transfers the synchronization tasks in the first database to the second database, and then synchronizes the data corresponding to the synchronization tasks from the first database and the second database to the third database in parallel. In this way, parallel processing of data is achieved, thereby effectively improving data synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1 is a flow chart of a data synchronization method according to an embodiment;

[0037] Figure 2 This is a flowchart of synchronizing all data in a first database to a second database.

[0038] Figure 3 A schematic diagram of the process of incremental data synchronization in one embodiment;

[0039] Figure 4 is a flowchart of a data synchronization method in another embodiment;

[0040] Figure 5 is a structural block diagram of a data synchronization device in one embodiment;

[0041] Figure 6FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] In one embodiment, Figure 1 As shown, a data synchronization method is provided. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0044] Step 102: Obtain the current number of synchronization tasks in the first database.

[0045] The first database is the original database that provides data. For example, the first database can be a distributed TiDB database, or an Oracle database, an ACCESS database, an SQL database, etc. In one embodiment, the first database is a database set or a single database.

[0046] In one embodiment, the first database is used to store a data table, which is used to implement data storage. Optionally, the data table has an attribute identifier that indicates whether the changed data in the data table needs to be synchronized to a third database when the data changes. The third database is the target database for receiving data. Optionally, the third database includes, but is not limited to, a MySQL database, a Kafka database, an Amazon S3 database, a GCS database, an Azure BlobStorage database, etc.

[0047] Optionally, it is also possible to distinguish based on the database whether to synchronize the changed data to the third database when the data changes. In one embodiment, when the first database is a database set, the database identifier is used to distinguish whether to synchronize the changed data to the third database when the data changes in the database.

[0048] In one embodiment, when a data change occurs in a first database, it is first determined whether the data table or database where the data change occurred requires synchronization with a third database. If so, a synchronization task can be generated for each change, and the synchronization task can be used to synchronize data from the first database to the third database. Optionally, all change tasks for the same data table or database can be generated into a single synchronization task, or change tasks for multiple data tables or databases can be generated into a single synchronization task.

[0049] In one embodiment, a synchronization tool can be used to process synchronization tasks, thereby synchronizing data from the first database to the third database. The synchronization tool is a processing tool for processing synchronization tasks. Optionally, the synchronization tool can be TiCDC. TiCDC is a TiDB incremental data synchronization tool that is implemented by pulling the TiKV change log. It has the ability to restore data to a state consistent with any upstream TSO (Terms of Service), and provides an open data protocol (TiCDC Open Protocol) to support other systems to subscribe to data changes. TiCDC has a master node and a slave node. The master node extracts incremental data from TiKV to form a synchronization task, and assigns the synchronization task to the slave node to process the synchronization data. After receiving the synchronization task assigned by the master node, the slave node synchronizes the data from one database to another according to the received synchronization task.

[0050] In one embodiment, when using TiCDC to synchronize data from a first database to a third database, processing nodes can be added based on the number of synchronization tasks to improve synchronization efficiency. However, when the number of synchronization tasks exceeds a first threshold, processing efficiency cannot be improved by adding processing nodes. Therefore, the number of synchronization tasks in the first database can be limited to improve the processing efficiency of the synchronization tasks. The first threshold is the limit on the number of synchronization tasks that the synchronization tool can process without reducing the processing speed. For example, the first threshold can be 700.

[0051] Optionally, TiCDC can be a database-level TiCDC for synchronizing incremental data based on a database, or a table-level TiCDC for synchronizing incremental data based on a data table.

[0052] Step 104: When the current number of synchronization tasks in the first database is greater than a first threshold, create a second database according to the current number of synchronization tasks in the first database.

[0053] The second database is an intermediate database that receives data synchronized from the first database and synchronizes the received data to the third database. Optionally, the second database is a database of the same type as the first database, or a database of a different type.

[0054] In one embodiment, the number of the second database may be multiple or one.

[0055] Optionally, the number of second databases can be set based on the number of synchronization tasks in the first database. For example, if the number of synchronization tasks in the first database is 1000 and the first threshold is 700, then the number of second databases created can be 1; if the number of synchronization tasks in the first database is 2000 and the first threshold is 700, then the number of second databases created can be 2.

[0056] In one embodiment, the current number of synchronization tasks and the number of second databases to be created can be set and stored in advance. When the second database needs to be created, the number of second databases can be obtained by querying the corresponding relationship based on the current number of synchronization tasks in the first database. In one embodiment, the range interval of the number of synchronization tasks can be set in correspondence with the number of second databases to be created. When the number of second databases needs to be determined, the current number of synchronization tasks in the first database is used to determine the range interval, and then the range interval is used to determine the number of second databases. For example, the current number of synchronization tasks in the first database is 1300, and the range is 700 to 1400. At this time, one second database can be created.

[0057] Step 106: Synchronize the incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database.

[0058] The first number is the number of synchronization tasks transferred to the second database. Optionally, after the first number of synchronization tasks are transferred to the second database, the difference between the current number of synchronization tasks in the first database and the first number should be less than or equal to a first threshold, and the number of synchronization tasks in the first database is reduced to no more than the first threshold, thereby improving the efficiency of synchronizing data from the first database to the third database.

[0059] In one embodiment, the first number can be determined based on the current number of synchronization tasks in the first database and a first threshold value, so that after the first number of synchronization tasks are transferred to the second database, the difference between the current number of synchronization tasks in the first database and the first number is less than or equal to the first threshold value. Optionally, half of the current number can be determined as the first number based on the current number, or after the first number of synchronization tasks are synchronized to the second database, the number of synchronization tasks in the first database is the first threshold value, at which time the first number of synchronization tasks synchronized to the second database is the difference between the current number of synchronization tasks in the first database and the first threshold value. For example, if the current number of synchronization tasks in the first database is 1,000 and the first threshold value is 700, then the first number can be 500 or 300, etc.

[0060] In one embodiment, the first number of synchronization tasks may be synchronization tasks randomly selected from the first database, or may be obtained based on the time sequence of generating the synchronization tasks.

[0061] In one embodiment, before the step of synchronizing the incremental data corresponding to the synchronization task to the second database, the data synchronization method further includes: obtaining the full amount of data corresponding to the synchronization task in the first database; synchronizing the full amount of data corresponding to the synchronization task in the first database to the second database. Figure 2 As shown, the export tool can be used to export the full data in the first database to local storage, and then the import tool can be used to import the full data stored locally into the second database. The full data is the data stored in the first database, and the full data does not include incremental data. Optionally, the export tool can be a Dumpling library table data export tool, and the import tool can be a Lightning library table data import tool. Dumpling is a tool written in Go (also known as Golang, a statically strongly typed, compiled language) for exporting data from a database. Lightning is a tool for importing full data into a TiDB cluster at high speed.

[0062] In one embodiment, Figure 3 As shown, after the full data transfer is completed, the incremental data in the first database is synchronized to the second database using a synchronization tool. Optionally, the synchronization tool includes but is not limited to the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool. Using the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool can quickly synchronize the incremental data in the first database to the second database.

[0063] In one embodiment, when there are multiple second databases, the step of synchronizing the incremental data corresponding to the first number of synchronization tasks to the second databases includes: obtaining the number of databases in the second databases, evenly distributing the first number of synchronization tasks to the number of second databases; and synchronizing the incremental data corresponding to the synchronization tasks from the first database to the corresponding second databases. For example, if the first number is 500 and the number of second databases is 2, the number of synchronization tasks in each second database can be 250.

[0064] Optionally, after synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database, the first number of synchronization tasks in the first database can be deleted. Optionally, when synchronizing the synchronization tasks from the first database to the second database, the synchronization tasks can be deleted, or a task identifier can be set to identify whether the synchronization of the synchronization tasks has been achieved. After completing the synchronization of the first number of synchronization tasks, the synchronization tasks marked as completed can be deleted in batches at one time.

[0065] Step 108: Synchronize the incremental data corresponding to the synchronization task from the first database and the second database to the third database in parallel.

[0066] In one embodiment, Figure 3 As shown, after synchronizing the incremental data corresponding to the first number of synchronization tasks in the first database to the second database, the synchronization tool can be further used to synchronize the incremental data corresponding to the synchronization tasks from the first database and the second database to the third database.

[0067] In one embodiment, the first database and the second database each have a corresponding synchronization tool. If there are multiple second databases, each second database has a corresponding synchronization tool. Optionally, the synchronization tool is TiCDC. In one embodiment, TiCDC is a table-level TiCDC. By using the synchronization tool corresponding to the second database and the synchronization tool corresponding to the first database, the incremental data corresponding to the synchronization task can be synchronized from the second database and the first database to the third database in parallel, which can reduce the processing efficiency caused by the reduction in synchronization speed due to the excessive number of synchronization tasks in a single database.

[0068] In the above data synchronization method, when the current number of synchronization tasks in the first database is greater than the first threshold, a new second database is created, the synchronization tasks in the first database are transferred to the second database, and then the data corresponding to the synchronization tasks are synchronized from the first database and the second database to the third database in parallel. By implementing parallel processing of data in this way, the number of synchronization tasks in the same database can be made less than or equal to the first threshold, reducing the problem of reduced efficiency caused by the reduction in synchronization speed due to the excessive number of synchronization tasks in the database. In addition, synchronizing data in parallel with multiple databases can further effectively improve data synchronization efficiency.

[0069] In one embodiment, after the step of synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database, the data synchronization method further includes: updating the number of synchronization tasks in the first database; when the number of synchronization tasks in the updated first database is greater than the first threshold, creating a new fourth database; synchronizing the incremental data corresponding to the second number of synchronization tasks to the fourth database, and deleting the second number of synchronization tasks that have been synchronized in the first database, wherein the difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold; and synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database, and the fourth database to the third database in parallel.

[0070] The fourth database is an intermediate database for receiving data from the first database and synchronizing it with the third database. Optionally, the fourth database can be of the same type as the first database, or a different type. In one embodiment, the fourth database can be a set of databases or a single database.

[0071] In one embodiment, the number of fourth databases can be determined based on the number of synchronization tasks in the updated first database and the first threshold. Optionally, the number of synchronization tasks can be set and stored in correspondence with the number of second databases to be created in advance. When it is necessary to create a second database, the number of second databases can be obtained by querying the corresponding relationship by obtaining the number of synchronization tasks in the updated first database. In one embodiment, the range interval of the number of synchronization tasks can be set in correspondence with the number of second databases to be created. When it is necessary to determine the number of second databases, the range interval is determined using the number of synchronization tasks in the updated first database, and then the range interval is used to determine the number of second databases. For example, the current number of synchronization tasks in the first database is 1300, and the range is 700 to 1400. At this time, one fourth database can be created.

[0072] The second number is the number of synchronization tasks transferred from the first database to the fourth database. In one embodiment, the second number can be determined based on the number of synchronization tasks in the updated first database and the first threshold. Optionally, the second number can be determined as half of the number of synchronization tasks in the updated first database based on the number of synchronization tasks in the updated first database. Alternatively, after synchronizing the second number of synchronization tasks to the fourth database, the number of synchronization tasks in the first database is the first threshold, and at this time, the second number of synchronization tasks synchronized to the fourth database is the difference between the number of synchronization tasks in the updated first database and the first threshold. For example, if the number of synchronization tasks in the updated first database is 1,000 and the first threshold is 700, then the first number can be 500 or 300, etc.

[0073] In one embodiment, the second number of synchronization tasks may be synchronization tasks randomly selected from the first database, or may be obtained based on the time sequence of generating the synchronization tasks.

[0074] After transferring the second number of synchronization tasks from the first database to the fourth database, the second number of synchronization tasks in the first database may be deleted. Optionally, the synchronization tasks that have completed synchronization may be deleted from the first database while synchronizing the second number of synchronization tasks. Alternatively, a task identifier may be set to identify whether synchronization of the synchronization tasks has been completed. After completing synchronization of the first number of synchronization tasks, the synchronization tasks marked as completed may be deleted in batches at one time.

[0075] In one embodiment, the first database, the second database, and the fourth database each have a corresponding synchronization tool. If there are multiple fourth databases, each fourth database has a corresponding synchronization tool. Optionally, the synchronization tool is TiCDC. By using the synchronization tool corresponding to the fourth database, the synchronization tool corresponding to the first database, and the synchronization tool corresponding to the second database, the incremental data corresponding to the synchronization task is synchronized from the first database, the second database, and the fourth database to the third database in parallel, which can reduce the processing efficiency caused by the reduction in synchronization speed due to the excessive number of synchronization tasks in a single database.

[0076] In one embodiment, the full amount of data corresponding to the synchronization task in the first database can be obtained, the full amount of data can be synchronized to the local storage using an export tool, and the full amount of data can be transferred from the local storage to the fourth database using an import tool. Optionally, the export tool can be a Dumpling library table data export tool, and the import tool can be a Lightning library table data import tool. After the synchronization of the full amount of data is achieved, the incremental data corresponding to the second amount of synchronization tasks is synchronized from the first database to the fourth database using a synchronization tool. Optionally, the synchronization tool includes but is not limited to the TiCDC library level synchronization tool, the Drainer library level synchronization tool, etc. The use of the TiCDC library level synchronization tool and the Drainer library level synchronization tool can achieve rapid synchronization of the incremental data in the first database to the fourth database.

[0077] In one embodiment, a synchronization tool may be used to synchronize only the second amount of incremental data to the fourth database. Optionally, the synchronization tool includes, but is not limited to, the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool. Using the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool can quickly synchronize the incremental data in the first database to the fourth database, thereby improving efficiency.

[0078] In this embodiment, after synchronizing the first number of synchronization tasks to the second database, the number of synchronization tasks in the first database is updated. When the number of synchronization tasks in the updated first database is greater than the first threshold, a fourth database can be further created, and the second number of synchronization tasks can be synchronized from the first database to the fourth database. Then, the incremental data corresponding to the synchronization tasks can be synchronized from the first database, the second database, and the fourth database in parallel to the third database, so that the number of synchronization tasks for each database does not exceed the first threshold, thereby improving the synchronization efficiency. At the same time, the incremental data can be synchronized to the third database by using the first database, the second database, and the fourth database in parallel, thereby effectively improving the efficiency of data synchronization.

[0079] In one embodiment, the above-mentioned data synchronization method also includes: obtaining the number of synchronization tasks in the second database; creating a fifth database when the number of synchronization tasks in the second database is greater than a first threshold; synchronizing the incremental data corresponding to the third number of synchronization tasks from the first database to the fifth database, and deleting the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0080] The fifth database is an intermediate database for receiving data from the first database and synchronizing it with the third database. Optionally, the fifth database can be of the same type as the first database, or a different type. In one embodiment, the fifth database can be a set of databases or a single database.

[0081] In one embodiment, the number of the fifth database can be determined based on the number of synchronization tasks in the updated first database and the first threshold. Optionally, the number of synchronization tasks can be set and stored in correspondence with the number of fifth databases to be created in advance. When the fifth database needs to be created, the number of fifth databases can be obtained by querying the corresponding relationship by obtaining the current number of synchronization tasks in the first database. In one embodiment, the range interval of the number of synchronization tasks can be set in correspondence with the number of fifth databases to be created. When the number of fifth databases needs to be determined, the current number of synchronization tasks in the first database is used to determine the range interval, and then the range interval is used to determine the number of fifth databases. For example, the current number of synchronization tasks in the first database is 1300, and the interval is 700 to 1400. At this time, one fourth database can be created. In one embodiment, the current number of synchronization tasks in the first database is the number of synchronization tasks remaining in the first database after the first number of synchronization tasks in the first database are synchronized to the second database.

[0082] The third number is the number of synchronization tasks transferred from the first database to the fifth database. In one embodiment, the third number can be determined based on the number of synchronization tasks in the first database and the first threshold. Optionally, the third number can be determined as half of the number of synchronization tasks in the first database based on the number of synchronization tasks in the first database. Alternatively, after synchronizing the third number of synchronization tasks to the fifth database, the number of synchronization tasks in the first database is the first threshold. At this time, the third number of synchronization tasks synchronized to the fifth database is the difference between the number of synchronization tasks in the first database and the first threshold. For example, if the number of synchronization tasks in the first database is 1,000 and the first threshold is 700, then the third number can be 500 or 300, etc.

[0083] In one embodiment, the third number of synchronization tasks may be synchronization tasks randomly selected from the first database, or may be obtained based on the time sequence of generating the synchronization tasks.

[0084] After transferring the third number of synchronization tasks from the first database to the fifth database, the third number of synchronization tasks in the first database can be deleted, wherein the current number of synchronization tasks in the first database minus the number of synchronization tasks synchronized to the second database and the number of synchronization tasks after the third number is less than or equal to the first threshold. Optionally, synchronization tasks that have completed synchronization can be deleted from the first database while synchronizing the third number of synchronization tasks, or a task identifier can be set to identify whether the synchronization of the synchronization tasks has been completed. After the synchronization of the third number of synchronization tasks is completed, the synchronization tasks marked as having completed synchronization can be deleted in batches at one time.

[0085] In one embodiment, the first database, the second database, and the fifth database each have a corresponding synchronization tool. If there are multiple fifth databases, each fifth database has a corresponding synchronization tool. Optionally, the synchronization tool is TiCDC. By using the synchronization tool corresponding to the fifth database, the synchronization tool corresponding to the first database, and the synchronization tool corresponding to the second database, the incremental data corresponding to the synchronization task is synchronized from the first database, the second database, and the fifth database to the third database in parallel, which can reduce the processing efficiency caused by the reduction in synchronization speed due to the excessive number of synchronization tasks in a single database.

[0086] In one embodiment, the full amount of data corresponding to the synchronization task in the first database can be obtained, the full amount of data can be synchronized to the local storage using an export tool, and the full amount of data can be transferred from the local storage to the fifth database using an import tool. Optionally, the export tool can be a Dumpling library table data export tool, and the import tool can be a Lightning library table data import tool. After the synchronization of the full amount of data is achieved, the incremental data corresponding to the third amount of synchronization tasks is synchronized from the first database to the fifth database using a synchronization tool. Optionally, the synchronization tool includes but is not limited to the TiCDC library level synchronization tool, the Drainer library level synchronization tool, etc. The use of the TiCDC library level synchronization tool and the Drainer library level synchronization tool can achieve rapid synchronization of the incremental data in the first database to the fifth database.

[0087] In one embodiment, a synchronization tool may be used to synchronize only the third amount of incremental data to the fifth database. Optionally, the synchronization tool includes, but is not limited to, the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool. Using the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool can quickly synchronize the incremental data in the first database to the fifth database, thereby improving efficiency.

[0088] In this embodiment, after synchronizing the first number of synchronization tasks to the second database, the number of synchronization tasks in the second database is determined. When the number of synchronization tasks in the second database is greater than the first threshold, a fifth database can be further created, and the third number of synchronization tasks can be synchronized from the first database to the fifth database. Then, the incremental data corresponding to the synchronization tasks are synchronized from the first database, the second database, and the fifth database to the third database in parallel, so that the number of synchronization tasks for each database does not exceed the first threshold, thereby improving the synchronization efficiency. At the same time, the incremental data is synchronized to the third database by using the first database, the second database, and the fifth database to effectively improve the efficiency of data synchronization.

[0089] In one embodiment, after the step of creating the fifth database when the number of synchronization tasks in the second database is greater than the first threshold, the data synchronization method further includes: synchronizing the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and deleting the synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; and synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database, and the fifth database to the third database in parallel.

[0090] The fourth number is the number of synchronization tasks transferred from the second database to the fifth database. In one embodiment, the fourth number can be determined based on the number of synchronization tasks in the second database and the first threshold. Optionally, the fourth number can be determined as half of the number of synchronization tasks in the second database based on the number of synchronization tasks in the second database. Alternatively, after synchronizing the fourth number of synchronization tasks to the fifth database, the number of synchronization tasks in the second database is the first threshold. At this time, the fourth number of synchronization tasks synchronized to the fifth database is the difference between the number of synchronization tasks in the second database and the first threshold. For example, if the number of synchronization tasks in the second database is 1,000 and the first threshold is 700, then the fourth number can be 500 or 300, etc.

[0091] In one embodiment, the fourth number of synchronization tasks may be synchronization tasks randomly selected from the second database, or may be obtained based on the time sequence of generating the synchronization tasks.

[0092] After transferring the fourth number of synchronization tasks from the second database to the fifth database, the third number of synchronization tasks in the second database may be deleted. Optionally, the synchronization tasks that have completed synchronization may be deleted from the second database simultaneously with the synchronization of the fourth number of synchronization tasks. Alternatively, a task flag may be set to indicate whether synchronization of the synchronization tasks has been completed. After the synchronization of the fourth number of synchronization tasks is completed, the synchronization tasks marked as completed may be deleted in batches at once.

[0093] In one embodiment, the first database, the second database, and the fifth database each have a corresponding synchronization tool. If there are multiple fifth databases, each fifth database has a corresponding synchronization tool. Optionally, the synchronization tool is TiCDC. By using the synchronization tool corresponding to the fifth database, the synchronization tool corresponding to the first database, and the synchronization tool corresponding to the second database, the incremental data corresponding to the synchronization task is synchronized from the first database, the second database, and the fifth database to the third database in parallel, which can reduce the processing efficiency caused by the reduction in synchronization speed due to the excessive number of synchronization tasks in a single database.

[0094] In one embodiment, the full amount of data corresponding to the synchronization task in the first database can be obtained, the full amount of data can be synchronized to the local storage using an export tool, and the full amount of data can be transferred from the local storage to the fourth database using an import tool. Optionally, the export tool can be a Dumpling library table data export tool, and the import tool can be a Lightning library table data import tool. After the synchronization of the full amount of data is achieved, the incremental data corresponding to the fourth amount of synchronization task is synchronized from the second database to the fifth database using a synchronization tool. Optionally, the synchronization tool includes but is not limited to the TiCDC library-level synchronization tool, the Drainer library-level synchronization tool, etc. The TiCDC library-level synchronization tool and the Drainer library-level synchronization tool can be used to quickly synchronize the incremental data in the second database to the fifth database.

[0095] In one embodiment, a synchronization tool may be used to synchronize only the fourth amount of incremental data to the fifth database. Optionally, the synchronization tool includes, but is not limited to, the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool. Using the TiCDC library-level synchronization tool and the Drainer library-level synchronization tool can quickly synchronize the incremental data in the second database to the fifth database, thereby improving efficiency.

[0096] In this embodiment, after synchronizing the first number of synchronization tasks to the second database, the number of synchronization tasks in the second database is determined. When the number of synchronization tasks in the second database is greater than the first threshold, a fifth database can be further created, and the fourth number of synchronization tasks can be synchronized from the second database to the fifth database. Then, the incremental data corresponding to the synchronization tasks are synchronized from the first database, the second database, and the fifth database to the third database in parallel, so that the number of synchronization tasks for each database does not exceed the first threshold, thereby improving the synchronization efficiency. At the same time, the incremental data is synchronized to the third database by using the first database, the second database, and the fifth database to synchronize in parallel, effectively improving the efficiency of data synchronization.

[0097] In this embodiment, Figure 4As shown, a data synchronization method is provided. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0098] Step 202: Obtain the current number of synchronization tasks in the first database;

[0099] Step 204: Determine whether the current number of synchronization tasks in the first database is greater than a first threshold. When the current number of synchronization tasks in the first database is greater than the first threshold, create a second database according to the current number of synchronization tasks in the first database.

[0100] Step 206: Synchronize the incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0101] Step 208: Update the number of synchronization tasks in the first database. When the number of synchronization tasks in the updated first database is greater than the first threshold, create a fourth database, synchronize the incremental data corresponding to the second number of synchronization tasks from the first database to the fourth database, and delete the second number of synchronization tasks that have been synchronized in the first database. The difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold.

[0102] Step 210: Obtain the number of synchronization tasks in the second database. When the number of synchronization tasks in the second database is greater than a first threshold, create a fifth database, synchronize incremental data corresponding to a third number of synchronization tasks from the first database to the fifth database, and delete the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold.

[0103] Step 212: Synchronize the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fourth database, and delete the fourth number of synchronization tasks in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold;

[0104] Step 214 : Synchronize the incremental data corresponding to the synchronization task from the first database, the second database, the fourth database, and the fifth database to the third database in parallel.

[0105] In this embodiment, the number of synchronization tasks in the first database is obtained. When the number of synchronization tasks is greater than a first threshold, a second database is created based on the number of synchronization tasks, the incremental data of the first number of synchronization tasks is parallelized to the second database, and the first number of synchronization tasks that have completed synchronization in the first database are deleted, wherein the difference between the current number and the first number is less than or equal to the first threshold. The number of synchronization tasks in the first database is updated. If the number of synchronization tasks in the first database after the update is greater than the first threshold, a fourth database is created, the incremental data corresponding to the second number of synchronization tasks is synchronized to the fourth database, and the second number of synchronization tasks that have completed synchronization in the first database are deleted, wherein the difference between the number of synchronization tasks in the first database after the update and the second number is less than or equal to the first threshold. Get the number of synchronization tasks in the second database. When the number of synchronization tasks in the second database is greater than the first threshold, create a fifth database, synchronize the incremental data corresponding to the third number of synchronization tasks to the fifth database, and delete the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; synchronize the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and delete the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold. Synchronize the incremental data corresponding to the synchronization tasks from the first database, the second database, the fourth database, and the fifth database to the third database. Through the above steps, the number of synchronization tasks in a single database is made less than or equal to the first threshold, so as to improve the efficiency of synchronizing data in a single database, and multiple data are synchronized in parallel, which can effectively improve the efficiency of data synchronization.

[0106] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0107] Based on the same inventive concept, embodiments of the present application also provide a data synchronization device for implementing the aforementioned data synchronization method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more data synchronization device embodiments provided below can be found in the above-mentioned limitations of the data synchronization method and will not be repeated here.

[0108] In one embodiment, Figure 5 As shown, a data synchronization device 300 is provided, comprising: a task quantity determination module 302, a database creation module 304, a first synchronization module 306 and a second synchronization module 308, wherein:

[0109] The task quantity determination module 302 is used to obtain the current quantity of synchronization tasks in the first database;

[0110] A database creation module 304 is configured to create a corresponding number of second databases according to the current number of synchronization tasks in the first database when the current number of synchronization tasks in the first database is greater than a first threshold;

[0111] A first synchronization module 306 is configured to delete the first number of synchronization tasks in the first database by synchronizing incremental data corresponding to the first number of synchronization tasks to the second database, wherein the difference between the current number and the first number is less than or equal to a first threshold;

[0112] The second synchronization module 308 is configured to synchronize the incremental data corresponding to the synchronization task from the first database and the second database to the third database in parallel.

[0113] In one embodiment, when there are multiple second databases, the first synchronization module 306 is also used to obtain the number of second databases, and evenly distribute the first number of synchronization tasks to the number of second databases; and synchronize the incremental data corresponding to the synchronization tasks from the first database to the corresponding second databases.

[0114] In one embodiment, the task quantity determination module 302 is also used to update the number of synchronization tasks in the first database; the database creation module 304 is also used to create a new fourth database when the number of synchronization tasks in the updated first database is greater than the first threshold; the first synchronization module 306 is also used to synchronize the incremental data corresponding to the second number of synchronization tasks to the fourth database, and delete the second number of synchronization tasks that have been synchronized in the first database, wherein the difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold; the second synchronization module 308 is also used to synchronize the incremental data corresponding to the synchronization tasks from the first database, the second database and the fourth database to the third database in parallel.

[0115] In one embodiment, the task quantity determination module 302 is also used to obtain the number of synchronization tasks in the second database; the database creation module 304 is also used to create a fifth database when the number of synchronization tasks in the second database is greater than the first threshold; the first synchronization module 306 is also used to synchronize the incremental data corresponding to the third number of synchronization tasks from the first database to the fifth database, and delete the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; the second synchronization module 308 is also used to synchronize the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0116] In one embodiment, the first synchronization module 306 is also used to synchronize the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and delete the synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; the second synchronization module 308 is also used to synchronize the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0117] In one embodiment, the data synchronization module further includes: a third synchronization module, configured to obtain the full amount of data corresponding to the synchronization task in the first database; and synchronize the full amount of data corresponding to the synchronization task in the first database to the second database.

[0118] Each module in the above-mentioned data synchronization device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0119] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store synchronization data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0120] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0121] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0122] Get the current number of synchronization tasks in the first database;

[0123] When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database;

[0124] Synchronize incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0125] The incremental data corresponding to the synchronization task is synchronized from the first database and the second database to the third database in parallel.

[0126] In one embodiment, when the processor executes the computer program, it also implements the following steps: when there are multiple second databases, the incremental data corresponding to the first number of synchronization tasks are synchronized to the second database, including: obtaining the number of databases of the second database, and evenly distributing the first number of synchronization tasks to the number of second databases; synchronizing the incremental data corresponding to the synchronization tasks from the first database to the corresponding second databases.

[0127] In one embodiment, the processor also implements the following steps when executing the computer program: after synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database, the method also includes: updating the number of synchronization tasks in the first database; when the number of synchronization tasks in the updated first database is greater than the first threshold, creating a new fourth database; synchronizing the incremental data corresponding to the second number of synchronization tasks to the fourth database, and deleting the second number of synchronization tasks that have been synchronized in the first database, wherein the difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fourth database to the third database in parallel.

[0128] In one embodiment, when the processor executes the computer program, it also implements the following steps: obtaining the number of synchronization tasks in the second database; creating a fifth database when the number of synchronization tasks in the second database is greater than a first threshold; synchronizing the incremental data corresponding to the third number of synchronization tasks from the first database to the fifth database, and deleting the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0129] In one embodiment, the processor also implements the following steps when executing the computer program: when the number of synchronization tasks in the second database is greater than the first threshold, after creating the fifth database, the method also includes: synchronizing the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and deleting the synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0130] In one embodiment, when the processor executes the computer program, it also implements the following steps: before synchronizing the incremental data corresponding to the synchronization task to the second database, it also includes: obtaining the full data corresponding to the synchronization task in the first database; synchronizing the full data corresponding to the synchronization task in the first database to the second database.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0132] Get the current number of synchronization tasks in the first database;

[0133] When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database;

[0134] Synchronize incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0135] The incremental data corresponding to the synchronization task is synchronized from the first database and the second database to the third database in parallel.

[0136] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when there are multiple second databases, the incremental data corresponding to the first number of synchronization tasks are synchronized to the second database, including: obtaining the number of databases of the second database, and evenly distributing the first number of synchronization tasks to the number of second databases; synchronizing the incremental data corresponding to the synchronization tasks from the first database to the corresponding second databases.

[0137] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: after synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database, the method also includes: updating the number of synchronization tasks in the first database; when the number of synchronization tasks in the updated first database is greater than the first threshold, creating a new fourth database; synchronizing the incremental data corresponding to the second number of synchronization tasks to the fourth database, and deleting the second number of synchronization tasks that have been synchronized in the first database, wherein the difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fourth database to the third database in parallel.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the number of synchronization tasks in the second database; creating a fifth database when the number of synchronization tasks in the second database is greater than a first threshold; synchronizing the incremental data corresponding to the third number of synchronization tasks from the first database to the fifth database, and deleting the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the number of synchronization tasks in the second database is greater than the first threshold, after creating the fifth database, the method also includes: synchronizing the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and deleting the synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0140] In one embodiment, when the computer program is executed by the processor, it also implements the following steps: before synchronizing the incremental data corresponding to the synchronization task to the second database, it also includes: obtaining the full data corresponding to the synchronization task in the first database; synchronizing the full data corresponding to the synchronization task in the first database to the second database.

[0141] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0142] Get the current number of synchronization tasks in the first database;

[0143] When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database;

[0144] Synchronize incremental data corresponding to the first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein the difference between the current number and the first number is less than or equal to the first threshold;

[0145] The incremental data corresponding to the synchronization task is synchronized from the first database and the second database to the third database in parallel.

[0146] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when there are multiple second databases, the incremental data corresponding to the first number of synchronization tasks are synchronized to the second database, including: obtaining the number of databases of the second database, and evenly distributing the first number of synchronization tasks to the number of second databases; synchronizing the incremental data corresponding to the synchronization tasks from the first database to the corresponding second databases.

[0147] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: after synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database, the method also includes: updating the number of synchronization tasks in the first database; when the number of synchronization tasks in the updated first database is greater than the first threshold, creating a new fourth database; synchronizing the incremental data corresponding to the second number of synchronization tasks to the fourth database, and deleting the second number of synchronization tasks that have been synchronized in the first database, wherein the difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fourth database to the third database in parallel.

[0148] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the number of synchronization tasks in the second database; creating a fifth database when the number of synchronization tasks in the second database is greater than a first threshold; synchronizing the incremental data corresponding to the third number of synchronization tasks from the first database to the fifth database, and deleting the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0149] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the number of synchronization tasks in the second database is greater than the first threshold, after creating the fifth database, the method also includes: synchronizing the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and deleting the synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; synchronizing the incremental data corresponding to the synchronization tasks from the first database, the second database and the fifth database to the third database in parallel.

[0150] In one embodiment, when the computer program is executed by the processor, it also implements the following steps: before synchronizing the incremental data corresponding to the synchronization task to the second database, it also includes: obtaining the full data corresponding to the synchronization task in the first database; synchronizing the full data corresponding to the synchronization task in the first database to the second database.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0152] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0153] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Get the current number of synchronization tasks in the first database; When the current number of synchronization tasks in the first database is greater than a first threshold, creating a second database according to the current number of synchronization tasks in the first database; Synchronize incremental data corresponding to a first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein a difference between the current number and the first number is less than or equal to the first threshold; Synchronize the incremental data corresponding to the synchronization task from the first database and the second database to the third database in parallel; The method further comprises: Obtain the number of synchronization tasks in the second database; When the number of synchronization tasks in the second database is greater than the first threshold, creating a fifth database; Synchronize the incremental data corresponding to the third number of synchronization tasks from the first database to the fifth database, and delete the third number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; Synchronize the incremental data corresponding to the synchronization task from the first database, the second database and the fifth database to the third database in parallel; When the number of synchronization tasks in the second database is greater than the first threshold, after creating the fifth database, the method further includes: Synchronize the incremental data corresponding to the fourth number of synchronization tasks from the second database to the fifth database, and delete the synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein the difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; The incremental data corresponding to the synchronization task is synchronized in parallel from the first database, the second database, and the fifth database to the third database.

2. The method according to claim 1, characterized in that The method further comprises: When there are multiple second databases, synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database includes: Obtain the number of databases of the second database, and evenly distribute the first number of synchronization tasks to the number of databases and the second databases; Synchronize the incremental data corresponding to the synchronization task from the first database to the corresponding second databases.

3. The method according to claim 1 or 2, characterized in that After synchronizing the incremental data corresponding to the first number of synchronization tasks to the second database, the method further includes: Updating the number of synchronization tasks in the first database; When the number of synchronization tasks in the updated first database is greater than the first threshold, creating a fourth database; Synchronize the incremental data corresponding to the second number of synchronization tasks to the fourth database, and delete the second number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the number of synchronization tasks in the first database after the update and the second number is less than or equal to the first threshold; The incremental data corresponding to the synchronization task is synchronized in parallel from the first database, the second database, and the fourth database to the third database.

4. The method according to claim 1, wherein Before synchronizing the incremental data corresponding to the synchronization task to the second database, the method further includes: Obtaining the full amount of data corresponding to the synchronization task in the first database; Synchronize all data corresponding to the synchronization task in the first database to the second database.

5. A data synchronization device, characterized in that: The device comprises: A task quantity determination module, configured to obtain the current quantity of synchronization tasks in the first database; a database creation module, configured to create a corresponding number of second databases according to the current number of synchronization tasks in the first database when the current number of synchronization tasks in the first database is greater than a first threshold; A first synchronization module is configured to synchronize incremental data corresponding to a first number of synchronization tasks to the second database, and delete the first number of synchronization tasks in the first database, wherein a difference between the current number and the first number is less than or equal to a first threshold; A second synchronization module is used to synchronize the incremental data corresponding to the synchronization task from the first database and the second database to a third database in parallel; The task quantity determination module is further configured to obtain the quantity of synchronization tasks in the second database; The database creation module is further configured to create a fifth database when the number of synchronization tasks in the second database is greater than the first threshold; The first synchronization module is further configured to synchronize incremental data corresponding to a third number of synchronization tasks from the first database to the fifth database, and delete the third number of synchronization tasks in the first database that have completed synchronization, wherein a difference between the current number and the number of synchronization tasks in the second database and the third number is less than or equal to the first threshold; The second synchronization module is further configured to synchronize the incremental data corresponding to the synchronization task from the first database, the second database, and the fifth database to the third database in parallel; The first synchronization module is further configured to synchronize incremental data corresponding to a fourth number of synchronization tasks from the second database to the fifth database, and delete synchronization tasks corresponding to the fourth number of synchronization tasks that have completed synchronization in the second database, wherein a difference between the number of synchronization tasks in the second database and the fourth number is less than or equal to the first threshold; The second synchronization module is further configured to synchronize the incremental data corresponding to the synchronization task from the first database, the second database, and the fifth database to the third database in parallel.

6. The device according to claim 5, characterized in that When there are multiple second databases, the first synchronization module is also used to obtain the number of databases of the second databases, and evenly distribute the first number of synchronization tasks to the number of second databases; and synchronize the incremental data corresponding to the synchronization tasks from the first database to the corresponding second databases.

7. The device according to claim 5, characterized in that The task quantity determination module is further configured to update the quantity of synchronization tasks in the first database; The database creation module is further configured to create a new fourth database when the number of synchronization tasks in the updated first database is greater than the first threshold; The first synchronization module is further configured to synchronize the incremental data corresponding to the second number of synchronization tasks to the fourth database, and delete the second number of synchronization tasks that have completed synchronization in the first database, wherein the difference between the number of synchronization tasks in the updated first database and the second number is less than or equal to the first threshold; The second synchronization module is further configured to synchronize the incremental data corresponding to the synchronization task from the first database, the second database, and the fourth database to the third database in parallel.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images