Data synchronization method, device and equipment, readable storage medium and program product

Abstract

The invention relates to a data synchronization method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining to-be-synchronized data of a source database; distributing the data to be synchronized; and writing the distributed to-be-synchronized data into the target database through at least one concurrent writing strategy according to the attribute of the target database. By adopting the method, the efficiency of writing the data into the target database and the data synchronization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data synchronization method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of database technology, there are many types of databases on the market, and many functions can be realized. Users can choose the corresponding database to use according to specific business needs. When business needs change, data needs to be migrated to other types of databases. The scenarios involve OLTP (On-Line Transaction Processing) to OLTP, OLTP to OLAP (On-Line Analytical Processing), OLTP to big data and other homogeneous or heterogeneous scenarios.

[0003] In traditional technology, data in the source database is distributed according to the dimensions of the table and processed by one writing thread, which results in low efficiency in writing to the target database. Summary of the invention

[0004] Based on this, it is necessary to provide a data synchronization method, apparatus, computer device, computer-readable storage medium and computer program product that can improve writing efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data synchronization method, comprising:

[0006] Get the data to be synchronized from the source database;

[0007] Distributing the data to be synchronized;

[0008] According to the properties of the target database, the distributed data to be synchronized is written into the target database through at least one concurrent writing strategy.

[0009] In one embodiment, writing the distributed data to be synchronized into the target database by at least one concurrent write strategy according to the properties of the target database includes:

[0010] If the target database supports splitting the transaction into at least one data table, the distributed data to be synchronized is written into the target database through a table-level concurrent write strategy.

[0011] In one embodiment, writing the distributed data to be synchronized into the target database through a table-level concurrent write strategy includes:

[0012] Decomposing the objects corresponding to the distributed data to be synchronized to obtain at least one data table;

[0013] Enqueue the to-be-synchronized data corresponding to the same data table identifier in the at least one data table into the corresponding write queue;

[0014] The dequeued data of the write queue is written into the target database in sequence.

[0015] In one embodiment, if the target database supports data rewriting, before sequentially writing the dequeued data of the write queue into the target database, the method further includes:

[0016] Rewriting the data in the write queue according to a preset rewriting rule to obtain first rewritten data;

[0017] Merging the first rewritten data to obtain first rewritten merged data;

[0018] The first rewritten merged data is dequeued to obtain dequeued data of the write queue.

[0019] In one embodiment, writing the distributed data to be synchronized into the target database by at least one concurrent write strategy according to the properties of the target database includes:

[0020] If the target database supports data rewriting, the distributed data to be synchronized is written into the target database through the rewriting merge strategy.

[0021] In one embodiment, writing the distributed data to be synchronized into the target database by rewriting the merge strategy includes:

[0022] rewriting the distributed data to be synchronized according to a preset rewriting rule to obtain second rewritten data;

[0023] Merging the second rewritten data to obtain second rewritten merged data;

[0024] The second rewritten merged data is written into the target database.

[0025] In one embodiment, if the target database supports decomposing a transaction into at least one data table, rewriting the distributed data to be synchronized according to a preset rewriting rule to obtain second rewritten data includes:

[0026] Decomposing the objects corresponding to the distributed data to be synchronized to obtain at least one data table;

[0027] rewriting the to-be-synchronized data corresponding to the same data table identifier in the at least one data table according to a preset rewriting rule to obtain the second rewritten data;

[0028] The step of writing the second rewritten merged data into the target database includes:

[0029] The second rewritten merged data is queued into a corresponding write queue, and the dequeued data of the write queue is sequentially written into the target database.

[0030] In a second aspect, the present application further provides a data synchronization device, including:

[0031] A data acquisition module is used to acquire the data to be synchronized from the source database;

[0032] A data distribution module, used for distributing the data to be synchronized;

[0033] The data writing module is used to write the distributed data to be synchronized into the target database through at least one concurrent writing strategy according to the properties of the target database.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the data synchronization method provided in the first aspect when executing the computer program.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the data synchronization method provided in the first aspect are implemented.

[0036] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data synchronization method provided in the first aspect.

[0037] The above-mentioned data synchronization method, apparatus, computer equipment, computer-readable storage medium and computer program product obtain the data to be synchronized from the source database, distribute the data to be synchronized, write the distributed data to be synchronized into the target database according to the properties of the target database through at least one concurrent write strategy. Since the properties of the target database are taken into consideration, the distributed data to be synchronized is written into the target database using at least one concurrent write strategy, which can retain the atomicity of the transaction and achieve efficient writing of the data to be synchronized into the target database, thereby improving the writing efficiency of the target database. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] FIG1 is an application environment diagram of a data synchronization method in one embodiment;

[0040] FIG2 is a schematic diagram of a flow chart of a data synchronization method in one embodiment;

[0041] FIG3 is a schematic diagram of a flow chart of step 206 in one embodiment;

[0042] FIG4 is a schematic flow chart of a data synchronization method in another embodiment;

[0043] FIG5 is a schematic diagram of a flow chart of a data synchronization method in another embodiment;

[0044] FIG6 is a schematic diagram of a flow chart of a data synchronization method in another embodiment;

[0045] FIG7 is a schematic flow chart of a data synchronization method in another embodiment;

[0046] FIG8 is a structural block diagram of a data synchronization device in one embodiment;

[0047] FIG9 is a structural block diagram of a data synchronization device in another embodiment;

[0048] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] MySQL is a relational database management system. Its master-slave architecture is one of the most commonly used database architectures, especially for websites with heavy loads. In the master-slave architecture, operations such as write, update, and delete performed by the master database will be synchronized to the slave database, so that the slave database will also execute the operations, thereby achieving data consistency in the master and slave databases.

[0051] In the process of data synchronization, the link that has the greatest impact on data synchronization efficiency is the write process to the target database. In order to achieve a smaller data synchronization delay and keep the data synchronized to the target database consistent with the data in the source database, the order of write operations on the target database must be consistent with the order of write operations in the source database. However, single-threaded data synchronization will inevitably fall short of the requirement. Therefore, we are required to maintain the order of changes to the same row of data and achieve concurrency of different rows of data during the write process of the target database.

[0052] The data synchronization method provided in the embodiment of the present application can be applied in the application environment shown in FIG1. ​​The data synchronization method provided in the embodiment of the present application can be executed by a data synchronization tool, that is, the data in the source database 102 is synchronized to the target database 104 through the data synchronization tool. The source database 102 and the target database 104 can be in a master-slave relationship, and the target database can include one or more. The data synchronization tool can be a terminal or a server.

[0053] In an exemplary embodiment, as shown in FIG2 , the data synchronization method includes the following steps 202 to 206. Among them:

[0054] Step 202: Acquire the data to be synchronized from the source database.

[0055] The data to be synchronized refers to the data that has not been synchronized to the target database. The data to be synchronized is usually the operation data in the source database, such as data for update, deletion, and insertion operations. The operation data of the source database needs to be synchronized to the target database so that the target database can also perform the same operations as the source database, thereby achieving data synchronization. Usually, the data to be synchronized of the source database can be obtained in chronological order or in order of arrangement.

[0056] Optionally, the data to be synchronized may be the full amount of data generated by the source database operating the stored data, or the incremental data generated within a preset period. For example, the log file of the source database may be obtained at intervals of a preset period, and the log file may be parsed to obtain the data to be synchronized. The preset period may be, for example, 0.5 seconds or 1 second, and the preset period may also include real-time. Among them, the log file of the source database may record operation information such as data writing, updating, and deleting. For example, the binary log of the MySQL database is a log file saved in binary form, which records the operation information such as writing, updating, and deleting of the corresponding data. Alternatively, the source database may capture changes in the database in real time through CDC (Change Data Capture) technology, such as adding, updating, and deleting operations, and record these changes. These changes may be synchronized to the data synchronization tool, and the data synchronization tool thereby obtains the data to be synchronized corresponding to the changes.

[0057] In an example, when the data to be synchronized in the source database reaches a preset size, the data to be synchronized in the source database can be obtained, so that the data to be synchronized can be synchronized, which can improve the synchronization efficiency and synchronization success rate.

[0058] Step 204: distribute the data to be synchronized.

[0059] Here, distribution can be understood as issuing or sending. The data to be synchronized can be distributed to each target database. Alternatively, synchronization tasks can be generated based on the data to be synchronized, and then the synchronization tasks can be distributed. For example, the transactions corresponding to the data to be synchronized can be disassembled to obtain different data tables, and then the data tables can be distributed to the corresponding write queues to be written into the target database.

[0060] In an optional application scenario, the data to be synchronized can be stored and distributed. If there are multiple target databases, the data to be synchronized is distributed to the data synchronization tools corresponding to each target database, so that the data to be synchronized is synchronized to the target database through the data synchronization tool. Alternatively, the data to be synchronized is distributed to the writing module of the data synchronization tool, and the data to be synchronized is written to the target database through the writing module.

[0061] Step 206: Write the distributed data to be synchronized into the target database using at least one concurrent write strategy according to the properties of the target database.

[0062] Among them, the concurrent write strategy refers to the strategy for concurrently writing to the target database. Concurrent write strategies may include, for example, table-level concurrent write strategies and rewrite merge strategies. Table-level concurrent write strategy refers to a strategy for concurrent writing based on a data table identifier, where the data table identifier refers to the identifier of the database table involved in the data to be synchronized. Rewrite merge strategy refers to a write strategy for rewriting and merging more data to obtain fewer data. Concurrency can be understood as multiple write channels writing data at the same time.

[0063] The properties of the target database refer to the properties of the operations supported by the target database. For example, if the database supports transaction decomposition, it means that the database has the property of supporting transaction decomposition. If the database supports data rewriting, it means that the database has the property of data rewriting, etc. It is easy to understand that if the target database has multiple properties, the distributed data to be synchronized can be written to the target database through multiple concurrent write strategies corresponding to each property. Among them, a transaction is a logical unit in the execution process of the database management system. A transaction consists of a finite sequence of database operations. Transactions have four basic characteristics, namely ACID characteristics: atomicity, consistency, isolation, and durability. Among them, atomicity means that all operations in the transaction are either completed or not executed. If an operation in the transaction fails, the entire transaction is rolled back (undone) and the database returns to the state before the transaction started.

[0064] Optionally, according to the properties of the target database, the distributed data to be synchronized may be written into the target database through a combination of multiple concurrent write strategies.

[0065] In the above data synchronization method, by obtaining the data to be synchronized from the source database and distributing the data to be synchronized, the distributed data to be synchronized is written into the target database through at least one concurrent write strategy according to the attributes of the target database. Since the attributes of the target database are taken into consideration, the distributed data to be synchronized is written into the target database using at least one concurrent write strategy, which can retain the atomicity of the transaction, and realize efficient writing of the data to be synchronized into the target database, thereby improving the writing efficiency of the target database.

[0066] In some embodiments, step 206 of writing the distributed data to be synchronized into the target database using at least one concurrent write strategy according to the properties of the target database includes:

[0067] If the target database supports splitting transactions into at least one data table, the distributed data to be synchronized is written to the target database through the table-level concurrent write strategy.

[0068] Among them, the table-level concurrent write strategy refers to a strategy for concurrent writing based on the data table identifier. The data table refers to the database table involved in the data to be synchronized, so the data table identifier refers to the identifier of the database table involved in the data to be synchronized. For example, the data to be synchronized includes an insert operation on object 1 in data table A, an update operation on object 2 in data table B, an insert operation on object 2 in data table C, etc., then data table A, data table B, and data table C refer to the database tables involved in the data to be synchronized. It is easy to understand that the number of database tables and the number of objects here are only exemplary. The number of database tables involved in actual application scenarios is not limited to three, and may include more. Different data tables correspond to different data table identifiers.

[0069] Optionally, if the target database supports decomposing a transaction into at least one data table, the transaction corresponding to the data to be synchronized is decomposed to obtain at least one data table, and the data to be synchronized corresponding to the at least one data table obtained by decomposition is queued into a write queue according to the data table identifier, and the dequeued data of the write queue is written into the target database. Exemplarily, one data table identifier may correspond to one write queue, or multiple data table identifiers may correspond to one write queue.

[0070] In this embodiment, when the target database supports decomposing transactions into at least one data table, the distributed data to be synchronized is written into the target database through a table-level concurrent write strategy to implement concurrent writing according to the data table, that is, the data to be synchronized corresponding to multiple data table identifiers can be written into the target database at the same time, thereby improving the efficiency of data writing and enabling the data written into the target database to maintain table-level sequentiality, thereby ensuring data consistency with the source database.

[0071] In an exemplary embodiment, the distributed data to be synchronized is written into the target database through a table-level concurrent write strategy, including:

[0072] The distributed transactions corresponding to the data to be synchronized are disassembled to obtain at least one data table; the data to be synchronized corresponding to the same data table identifier in at least one data table are queued into the corresponding write queue; and the dequeued data of the write queue are written into the target database in sequence.

[0073] In this embodiment, one data table identifier corresponds to one write queue, and the data table identifier corresponding to the disassembled data table can be identified, and the data to be synchronized corresponding to the same data table identifier is queued into the write queue corresponding to the data table identifier, and then the data dequeued from the write queue is sequentially written into the target database in the order of dequeuing. It is easy to understand that if there are multiple data table identifiers, it is possible to simultaneously queue the data to be synchronized corresponding to multiple data table identifiers into the corresponding write queues, and then write the data dequeued from the multiple write queues into the target database at the same time, thereby improving the writing efficiency. Among them, the transaction corresponding to the distributed data to be synchronized can be one transaction or multiple different transactions. In this embodiment, each transaction is decomposed separately to obtain at least one data table.

[0074] Optionally, for the data to be synchronized corresponding to the same data table identifier, different rows of data for the same object can be merged into one row of data. For example, the data to be synchronized corresponding to the same data table identifier includes an insert operation on object 1 in data table A and also includes a change operation on object 1 in data table A. Then these two rows of data can be merged into one row of data, that is, an insert operation and a change operation are performed on object 1 in data table A. When merging, it is still necessary to merge according to the operation sequence, that is, the operation sequence of the merged data is consistent with that before merging. Then the merged data is queued into the corresponding write queue.

[0075] In an example, as shown in FIG3 , the distributed data to be synchronized include “Table C ID1 Insert”, “Table C ID2 Insert”, “Table A ID2 Insert”, “Table B ID2 Delete”, “Table A ID1 Insert”, and “Table B ID2 Update”, where “Table C ID1 Insert” indicates a delete operation on object 1 in Table C, and the meanings of other data to be synchronized are similar and are not described in detail here. The things corresponding to the distributed data to be synchronized are disassembled to obtain data tables corresponding to data table identifier A, data table identifier B, and data table identifier C. For data table identifier A, the data to be synchronized “Table A ID2 Insert” and “Table A ID1 Insert” corresponding to data table identifier A are queued into the corresponding write queue, i.e., the table A execution queue; for data table identifier B, the data to be synchronized “Table B ID2 Delete” and “Table B ID2 Update” corresponding to data table identifier B are queued into the corresponding write queue, i.e., the table B execution queue; for data table identifier C, the data to be synchronized “Table C ID1 Insert” and “Table C ID2 Insert” corresponding to data table identifier C are queued into the corresponding write queue, i.e., the table C execution queue.

[0076] In this embodiment, by disassembling the things corresponding to the distributed data to be synchronized, at least one data table is obtained, and the data to be synchronized corresponding to the same data table identifier in at least one data table is queued into the corresponding write queue, and the dequeued data of the write queue is written into the target database in sequence, the data to be synchronized can be accurately and concurrently written into the target database in the order of the table level, thereby improving the database writing efficiency and the accuracy of table-level concurrency.

[0077] In some embodiments, if the target database supports data rewriting, before sequentially writing the dequeued data of the write queue into the target database, the process further includes:

[0078] Rewrite the data in the write queue according to a preset rewrite rule to obtain first rewritten data; merge the first rewritten data to obtain first rewritten merged data; and dequeue the first rewritten merged data to obtain dequeue data of the write queue.

[0079] Among them, data rewriting refers to converting a DML (Data Manipulation Language) statement into another equivalent but possibly more optimized or more suitable for specific needs DML statement. The data rewriting method is determined by the type of the target database, that is, different types of target databases may support different data rewriting methods. Once the target database is determined, the corresponding rewriting method is also determined. The data rewriting method can be characterized by a preset rewriting rule, that is, the preset rewriting rule is used to characterize the data rewriting method.

[0080] Optionally, the data in the write queue is rewritten according to a preset rewrite rule to obtain first rewritten data, and then the first rewritten data is merged, for example, multiple lines of operation data of the same object are simplified into one line of operation data to obtain first rewritten merged data, the first rewritten merged data is dequeued to obtain dequeued data of the write queue, and then the dequeued data of the write queue is sequentially written into the target database. Exemplarily, as shown in FIG4, based on the example shown in FIG3, the data in each write queue is dequeued after being processed by the rewrite merge strategy to obtain dequeued data, and then the dequeued data of each write queue is concurrently written into the target database. It is possible to further rewrite and merge data in the write queue, while ensuring data consistency and integrity, reducing the amount of data in the write queue, and further improving data writing efficiency.

[0081] In one example, as shown in FIG. 5 , the data synchronization method includes the following steps 502 to 512 .

[0082] Step 502: Decompose the distributed transactions corresponding to the data to be synchronized to obtain at least one data table.

[0083] Step 504: Enqueue the to-be-synchronized data corresponding to the same data table identifier in at least one data table into a corresponding write queue.

[0084] Step 506: rewrite the data in the write queue according to a preset rewriting rule to obtain first rewritten data.

[0085] Step 508: merge the first rewritten data to obtain first rewritten merged data.

[0086] Step 510: dequeue the first rewritten merged data to obtain dequeue data of the write queue.

[0087] Step 512, write the dequeued data of the write queue into the target database in sequence.

[0088] In this embodiment, by disassembling the things corresponding to the data to be synchronized, at least one data table is obtained, the data to be synchronized corresponding to the same data table identifier is queued into the corresponding write queue, and then the data in the write queue is rewritten and merged according to the preset rewrite rule, the rewritten and merged data is dequeued, and then the dequeued data is written into the target database in sequence, so as to combine the table-level concurrency strategy and the rewrite and merge strategy to realize concurrent data writing, which can better adapt to the properties of the target database, reduce synchronization delay, and further improve data writing throughput and writing efficiency.

[0089] In some embodiments, step 206 of writing the distributed data to be synchronized into the target database using at least one concurrent write strategy according to the properties of the target database includes:

[0090] If the target database supports data rewriting, the distributed data to be synchronized is written to the target database through the rewriting merge strategy.

[0091] Data rewriting refers to converting a DML statement into another equivalent but possibly more optimized or more suitable for specific needs. The premise of data rewriting is to ensure consistency before and after rewriting and integrity after rewriting. Rewriting and merging strategies refer to strategies for reducing the number of data entries by rewriting and merging, that is, strategies for reducing the number of data entries through rewriting and merging strategies and then writing them into the target database.

[0092] Optionally, the distributed data to be synchronized can be rewritten and merged according to the preset rewriting rules, and the rewritten and merged data can be written to the target database through batch submission. In database operations, batch submission allows multiple SQL statements to be combined into a batch for execution, rather than sending SQL statements to the database one by one for execution, which can significantly reduce the number of database connections and transaction submissions, thereby improving the performance of data operations.

[0093] In this embodiment, when the target database supports data rewriting, the distributed data to be synchronized is written into the target database through a rewrite merge strategy, so that the amount of synchronized data can be reduced while ensuring data consistency and integrity, thereby improving synchronization efficiency.

[0094] In an exemplary embodiment, writing the distributed data to be synchronized into the target database by rewriting the merge strategy includes:

[0095] The distributed data to be synchronized is rewritten according to a preset rewriting rule to obtain second rewritten data; the second rewritten data is merged to obtain second rewritten merged data; and the second rewritten merged data is written into a target database.

[0096] If the target database supports data rewriting, a preset rewriting rule is determined according to the rewriting attribute of the target database. The distributed data to be synchronized is rewritten according to the preset rewriting rule to obtain second rewritten data, and the second rewritten data is merged to obtain second rewritten merged data. The second rewritten merged data can be submitted in batches to achieve concurrent writing to the target database and improve data writing efficiency.

[0097] Exemplarily, the data to be synchronized is rewritten according to preset rewriting rules. For example, the update operation of a single row can be rewritten into a delete operation and an insert operation, thereby reducing the operation types at the DML level. That is, the update operation can be removed from the DML operations of update, insert and delete, and the data can be merged based on the rewritten data, that is, the operations of the same object in different rows are simplified into operations for one row. For example, if a row of data is updated first and then deleted, the deletion can be directly executed, and then the deletion operations can be submitted in batches. Similarly, the insertion operations can be submitted in batches to ensure the atomicity of the batch submission.

[0098] In this embodiment, the distributed data to be synchronized is rewritten and merged according to preset rewriting rules to obtain rewritten merged data, and then the rewritten merged data is written into the target database. The difficulty of merging can be reduced by data rewriting, and the amount of synchronized data can be reduced by merging, thereby improving synchronization efficiency while ensuring data consistency.

[0099] In an exemplary embodiment, if the target database supports decomposing a transaction into at least one data table, rewriting the distributed data to be synchronized according to a preset rewriting rule to obtain second rewritten data includes:

[0100] Disassembling the things corresponding to the distributed data to be synchronized to obtain at least one data table; rewriting the data to be synchronized corresponding to the same data table identifier in at least one data table according to a preset rewriting rule to obtain the second rewritten data;

[0101] Writing the second rewrite merged data to the target database includes:

[0102] The second rewrite merged data is queued into the corresponding write queue, and the dequeued data of the write queue is written into the target database in sequence.

[0103] Among them, the target database supports data rewriting and also supports the decomposition of transactions into at least one data table. Then, data writing to the target database can be realized by combining the rewrite merge strategy and the table-level concurrent write strategy. The transactions corresponding to the distributed data to be synchronized are decomposed to obtain at least one data table. The data to be synchronized corresponding to the same data table identifier in at least one data table is rewritten according to the preset rewrite rule corresponding to the target database to obtain the second rewritten data. The second rewritten data is then merged to obtain the second rewritten merged data. The second rewritten merged data is queued into the write queue corresponding to the corresponding data table identifier in a time sequence from early to late, and the dequeued data of the write queue is sequentially written into the target database.

[0104] Exemplarily, as shown in FIG. 6 , the data synchronization method includes the following steps 602 to 608 .

[0105] Step 602: Decompose the distributed transactions corresponding to the data to be synchronized to obtain at least one data table.

[0106] Step 604 : rewrite the to-be-synchronized data corresponding to the same data table identifier in at least one data table according to a preset rewriting rule to obtain second rewritten data.

[0107] Step 606: merge the second rewritten data to obtain second rewritten merged data.

[0108] Step 608: Enqueue the second rewritten merged data into the corresponding write queue, and write the dequeued data of the write queue into the target database in sequence.

[0109] In this embodiment, by combining the rewrite merge strategy and the table-level concurrent write strategy, the data to be synchronized corresponding to the same data table identifier is rewritten and merged according to the preset rewrite rules, and the rewritten and merged data is written to the target database through at least one queue, so that data concurrent writing can be achieved and data writing efficiency can be improved.

[0110] In an exemplary embodiment, the target database can support splitting a transaction into at least one data table and also supports data rewriting. As shown in FIG. 7 , the data synchronization method includes the following steps 702 to 710 .

[0111] Step 702: Decompose the distributed transactions corresponding to the data to be synchronized to obtain at least one data table.

[0112] Step 704: Enqueue the to-be-synchronized data corresponding to the same data table identifier in at least one data table into a corresponding write queue.

[0113] Step 706: rewrite the outgoing data of each write queue according to a preset rewriting rule to obtain third rewritten data.

[0114] Step 708: merge the third rewritten data to obtain third rewritten merged data.

[0115] Step 710: submit the third rewritten merged data in batches according to a preset quantity and write them into the target database.

[0116] In the above embodiment, by enqueuing the data to be synchronized corresponding to the same data table identifier into the write queue, and then rewriting and merging the dequeued data, and submitting them in batches to write to the target database, it is possible to better adapt to the properties of the target database, concurrently write the data to be synchronized through multiple write queues, and reduce the amount of dequeued data, further improving the efficiency of writing data to the target database. Compared with the synchronization capability of the single-threaded MySQL master-slave synchronization, it is possible to achieve concurrent data writing, which is more suitable for the writing characteristics of the target database, and the writing performance is improved several times.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a data synchronization device for implementing the data synchronization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data synchronization device embodiments provided below can refer to the limitations on the data synchronization method above, and will not be repeated here.

[0119] In an exemplary embodiment, as shown in FIG8 , a data synchronization device is provided, including: a data acquisition module 802, a data distribution module 804 and a data writing module 806, wherein:

[0120] The data acquisition module 802 is used to acquire the data to be synchronized from the source database;

[0121] A data distribution module 804, used for distributing the data to be synchronized;

[0122] The data writing module 806 is used to write the distributed data to be synchronized into the target database through at least one concurrent writing strategy according to the properties of the target database.

[0123] In some embodiments, the data writing module 806 is further configured to write the distributed data to be synchronized into the target database through a table-level concurrent writing strategy if the target database supports splitting a transaction into at least one data table.

[0124] In some embodiments, the data writing module 806 is also used to disassemble the objects corresponding to the distributed data to be synchronized to obtain at least one data table; queue the data to be synchronized corresponding to the same data table identifier in at least one data table into the corresponding write queue; and write the dequeued data of the write queue into the target database in sequence.

[0125] In some embodiments, the above-mentioned device also includes a data rewrite merge module, which is used to rewrite the data in the write queue according to a preset rewrite rule to obtain first rewritten data before writing the out-of-queue data of the write queue into the target database in sequence if the target database supports data rewrite; merge the first rewritten data to obtain first rewrite merged data; and dequeue the first rewrite merged data to obtain the out-of-queue data of the write queue.

[0126] In some embodiments, the data writing module 806 is further configured to write the distributed data to be synchronized into the target database through a rewrite merge strategy if the target database supports data rewrite.

[0127] In some embodiments, the data writing module 806 is further used to rewrite the distributed data to be synchronized according to a preset rewriting rule to obtain second rewritten data; merge the second rewritten data to obtain second rewritten merged data; and write the second rewritten merged data into the target database.

[0128] In some embodiments, if the target database supports decomposing transactions into at least one data table, the data writing module 806 is also used to decompose the transactions corresponding to the distributed data to be synchronized to obtain at least one data table; rewrite the data to be synchronized corresponding to the same data table identifier in at least one data table according to a preset rewrite rule to obtain second rewritten data; and queue the second rewritten merged data into the corresponding write queue, and write the dequeued data of the write queue into the target database in sequence.

[0129] In an optional embodiment, as shown in FIG9 , the data synchronization device includes a data acquisition module 902, a data distribution module 904, and a data writing module 906, wherein the data acquisition module 902 is used to acquire the data to be synchronized of the source database, the data distribution module 904 is used to store and distribute the data to be synchronized, and the stored data to be synchronized can be distributed to the data writing modules corresponding to each target database, and the data writing module 906 is used to write the data to be synchronized distributed by the data storage module into the target database, wherein the data writing module 906 can write the distributed data to be synchronized into the target database through at least one concurrent writing strategy according to the properties of the target database. For example, the data writing module 906 can pre-process the distributed data to be synchronized, such as data time series arrangement, denoising and other pre-processing, to obtain pre-processed data, and then write the pre-processed data into the target database through at least one of the concurrent writing strategies such as table-level concurrent writing strategy, rewrite merge strategy, etc., so as to realize data concurrent writing into the target database, and the writing strategy is better consistent with the properties of the target database, further improving the writing efficiency, reducing the delay, and improving the writing throughput.

[0130] Each module in the above data synchronization device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0131] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be synchronized. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0132] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the data synchronization method in the above embodiment when executing the computer program.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data synchronization method in the above embodiment are implemented.

[0135] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the data synchronization method in the above embodiment when executed by a processor.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Get the data to be synchronized from the source database; Distributing the data to be synchronized; According to the properties of the target database, the distributed data to be synchronized is written into the target database through at least one concurrent writing strategy.

2. The method according to claim 1, characterized in that The step of writing the distributed data to be synchronized into the target database by using at least one concurrent writing strategy according to the properties of the target database includes: If the target database supports splitting the transaction into at least one data table, the distributed data to be synchronized is written into the target database through a table-level concurrent write strategy.

3. The method according to claim 2, characterized in that The step of writing the distributed data to be synchronized into the target database through a table-level concurrent write strategy includes: Decomposing the objects corresponding to the distributed data to be synchronized to obtain at least one data table; Enqueue the to-be-synchronized data corresponding to the same data table identifier in the at least one data table into the corresponding write queue; The dequeued data of the write queue is written into the target database in sequence.

4. The method according to claim 3, characterized in that If the target database supports data rewriting, before sequentially writing the dequeued data of the write queue into the target database, the method further includes: Rewriting the data in the write queue according to a preset rewriting rule to obtain first rewritten data; Merging the first rewritten data to obtain first rewritten merged data; The first rewritten merged data is dequeued to obtain dequeued data of the write queue.

5. The method according to claim 1, characterized in that The step of writing the distributed data to be synchronized into the target database by using at least one concurrent writing strategy according to the properties of the target database includes: If the target database supports data rewriting, the distributed data to be synchronized is written into the target database through the rewriting merge strategy.

6. The method according to claim 5, characterized in that The step of writing the distributed data to be synchronized into the target database by rewriting the merge strategy includes: rewriting the distributed data to be synchronized according to a preset rewriting rule to obtain second rewritten data; Merging the second rewritten data to obtain second rewritten merged data; The second rewritten merged data is written into the target database.

7. The method according to claim 6, characterized in that If the target database supports decomposing a transaction into at least one data table, rewriting the distributed data to be synchronized according to a preset rewriting rule to obtain second rewritten data includes: Decomposing the objects corresponding to the distributed data to be synchronized to obtain at least one data table; rewriting the to-be-synchronized data corresponding to the same data table identifier in the at least one data table according to a preset rewriting rule to obtain the second rewritten data; The step of writing the second rewritten merged data into the target database includes: The second rewritten merged data is queued into a corresponding write queue, and the dequeued data of the write queue is sequentially written into the target database.

8. A data synchronization device, characterized in that: The device comprises: A data acquisition module is used to acquire the data to be synchronized from the source database; A data distribution module, used for distributing the data to be synchronized; The data writing module is used to write the distributed data to be synchronized into the target database through at least one concurrent writing strategy according to the properties of the target database.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images