Data synchronization method, device and equipment and readable storage medium

Abstract

The invention discloses a data synchronization method, device and equipment and a readable storage medium, and relates to the field of big data, the method comprises the following steps: creating an extensible markup language (XML) file, establishing a table mapping relation between a target table and at least two source tables through the XML file, and establishing a field mapping relation between the target table and the at least two source tables; and then, the XML file is configured into the canal component, so that the canal component performs corresponding addition, deletion and modification operation on the target table based on addition, deletion and modification operation of the source table, and the data synchronization rate is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data synchronization method, device, equipment and readable storage medium. Background Art

[0002] In an enterprise, data is usually persistently stored in the MySQL database of each business system. Given the limitations of MySQL in handling specific application scenarios, it is usually necessary to synchronize MySQL data in real time to other database systems such as Hive, TiDB, Kudu, HBase, ES, etc. to meet diverse business needs.

[0003] Alibaba's open source canal middleware cleverly uses MySQL's master-slave replication mechanism by disguising itself as a MySQL slave. This mechanism can capture the data change log binlog of the MySQL master database and send these change data to the message queue MQ in real time. Subsequently, these change data are distributed to various downstream database systems to achieve real-time data synchronization. The data synchronization time of this process can reach milliseconds or even sub-seconds.

[0004] However, the canal component currently does not support many-to-one data mapping, that is, it is not possible to directly map data from multiple source tables to a single target table. This is particularly inconvenient when converting online transaction processing (OLTP) tables to online analytical processing (OLAP) tables, because the design of OLAP tables often requires merging multiple related narrow tables into a wide table.

[0005] At present, in order to solve this problem, a feasible solution is to first perform a one-to-one mapping of multiple tables, and then perform multi-table association queries in the target database system. However, this method has significant performance bottlenecks, especially in the scenario of real-time data synchronization. Since the business tolerance for data delay is usually at the millisecond level, and multi-table association queries may take seconds or even minutes, this is obviously unacceptable. For example, suppose that each of the three tables A, B, and C has 1 million records, and there are about 10 changes per second every day. Although canal can synchronize these changes to the target system at a sub-second speed, it may take tens of seconds to perform association queries on these three tables with millions of records in the target system, which seriously violates the real-time requirements.

[0006] Therefore, how to improve the data synchronization rate under many-to-one mapping has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] In view of this, the main purpose of the present application is to provide a time synchronization method, apparatus, device and readable storage medium, the purpose of which is to increase the data synchronization rate.

[0008] The first aspect of the present application provides a data synchronization method, the method comprising:

[0009] Creating an extensible markup language XML file, the XML file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between the target table and at least two source tables;

[0010] Configure the XML file to the canal component so that the canal component can perform corresponding addition, deletion, and modification operations on the target table based on the addition, deletion, and modification operations of the source table.

[0011] In some implementations of the first aspect of the present application, the source table includes the following types: a main table and an auxiliary table.

[0012] In some implementations of the first aspect of the present application, the method includes:

[0013] In response to the deletion of a field in the source table, determining the type of the source table;

[0014] When the source table is the primary table, based on the mapping relationship between the target table fields and all the source table fields, the mapping fields corresponding to the deleted fields of the primary table are determined in the target table, and the mapping fields corresponding to the deleted fields of the primary table are deleted.

[0015] When the source table is an auxiliary table, according to the mapping relationship between the target table fields and all the fields of the source table, the mapping fields corresponding to the fields deleted from the auxiliary table are determined in the target table, and the values ​​of the mapping fields corresponding to the fields deleted from the auxiliary table are modified to null values.

[0016] In some implementations of the first aspect of the present application, the method further includes:

[0017] In response to the insertion of a field in a source table, determining the type of the source table;

[0018] When the source table is the primary table, according to the mapping relationship between the target table field and all the fields of the source table, the mapping field corresponding to the field inserted in the primary table is determined in the target table, and the insertion value of the field inserted in the primary table is inserted in the mapping field corresponding to the field inserted in the primary table;

[0019] When the source table is an auxiliary table, based on the mapping relationship between the target table fields and all the fields of the source table, the mapping fields corresponding to the fields inserted into the auxiliary table are determined in the target table, and the values ​​of the mapping fields corresponding to the fields inserted into the auxiliary table are modified to the insertion values ​​of the fields inserted into the auxiliary table.

[0020] In some implementations of the first aspect of the present application, the method further includes:

[0021] In response to the modification of the fields in the source table, based on the mapping relationship between the target table fields and all the fields in the source table, the mapping fields corresponding to the modified fields in the source table are determined in the target table, and the values ​​of the mapping fields corresponding to the modified fields in the auxiliary table are modified to the values ​​of the modified fields in the source table.

[0022] In some implementations of the first aspect of the present application, the XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of the source table, and mapping of target table fields to all fields of the source table.

[0023] In some implementations of the first aspect of the present application, performing corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations on the source table includes:

[0024] Based on the addition, deletion and modification operations of the source table, assemble the corresponding structured query statements and execute the structured query statements in the database to which the target table belongs.

[0025] A second aspect of the present application provides a data synchronization device, the device comprising:

[0026] A file creation module, used to create an extensible markup language XML file, the XML file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between the target table and at least two source tables;

[0027] The file configuration module is used to configure the XML file into the canal component so that the canal component can perform corresponding addition, deletion and modification operations on the target table based on the addition, deletion and modification operations of the source table.

[0028] A third aspect of the present application provides a data synchronization device, which includes a memory and a processor, and the processor is used to execute a program stored in the memory and run any data synchronization method provided in the first aspect of the present application.

[0029] The fourth aspect of the present application provides a readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, each step of any data synchronization method provided in the first aspect of the present application is implemented.

[0030] The technical solution provided by this application has the following beneficial effects:

[0031] In the technical solution provided in the present application, first, an extensible markup language XML file is created, and the XML1 file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between a target table and at least two source tables; then, the XML file is configured in the canal component, giving the Canal component the ability to map one target table to multiple source tables, thereby enabling the Canal component to directly perform corresponding add, delete, and modify operations on the target table according to the add, delete, and modify operations occurring in multiple source tables. It can be seen that this technical solution effectively avoids the problem of synchronizing multiple table data to the target system separately through one-to-one mapping, and then performing multi-table association queries in the target database, and the problem that the synchronization time increases with the increase of data volume, and has a higher data synchronization rate and reduces synchronization delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0033] Figure 2 A flowchart of another data synchronization method provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of a flow chart of a canal component provided in an embodiment of the present application performing corresponding operations on a target table according to the operation type of a source table;

[0035] Figure 4 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] In the canal component, you can implement single-table data mapping by configuring the XML file in the canal-adapter module. For example:

[0039] #Data source;

[0040] dataSourceKey:defaultDS

[0041] #Database mapping configuration section;

[0042] dbMapping:

[0043] #Specify the name of the source database;

[0044] database:mytest

[0045] #Specify the table name in the source database;

[0046] table:user

[0047] #Specify the table name in the target database;

[0048] targetTable:mytest2.user

[0049] #Primary key configuration of the target table

[0050] targetPk:

[0051] id:id

[0052] #Target field mapping configuration

[0053] targetColumns:

[0054] id:id

[0055] name:name

[0056] Among them, dataSourceKey and dbMapping.database represent the source database source and the target database source respectively; dbMapping.table and dbMapping.targetTable represent the source database table and the target database table respectively; dbMapping.targetPk represents the mapping between the target table fields and the primary key fields of the source table; dbMapping.targetColumns represents the mapping between the target table fields and all the fields of the source table.

[0057] It should be noted that the above is a one-to-one mapping implementation, while for a one-to-many mapping implementation, that is, one source table is mapped to multiple target tables, multiple XML files can be configured, wherein each XML file indicates the same source database table, different target data tables and different field mappings.

[0058] The canal component can parse the binary log binlog, combine the add, delete, and modify operations of the source table with the mapping file, and perform add, delete, and modify operations on the target table and fields.

[0059] For example, the source table S has fields T1, T2, and T3, and the target table T has corresponding fields T11, T22, and T33. When the structured query SQL statement for adding operations to the source table is:

[0060] insert S(T1,T2,T3)values(V1,V2,V3);

[0061] Then, the SQL statement for the canal component to add operations to the target table is:

[0062] insert T(T11,T22,T33)values(V1,V2,V3);

[0063] It can be understood that the deletion and modification operations are the same as above.

[0064] Currently, the canal component itself only supports one-to-one mapping, and does not provide many-to-one mapping implementation, that is, mapping multiple source tables to one target table. In actual work, when converting OLTP tables to OLAP tables, multiple narrow tables need to be mapped to one wide table. Therefore, when designing OLTP tables, multiple tables should be split as atomically as possible. When designing OLAP tables, data with associations should be designed into one table as much as possible. This natural design difference leads to many many-to-one data synchronization mapping requirements. Under the existing mechanism, a feasible way is to first do one-to-one mapping of multiple tables, and then associate multiple tables in the target database system. However, this implementation method has a big flaw, that is, there is a low performance problem in multi-table association. Especially when canal is applied in real-time data synchronization scenarios, the acceptable range of data delay for the business is milliseconds, and the association of multiple tables, especially large tables, will have several seconds or even minutes of association calculation, which is unacceptable to the business.

[0065] In view of this, embodiments of the present application provide a data synchronization method, apparatus, device, and readable storage medium, the purpose of which is to increase the data synchronization rate and reduce the data synchronization delay.

[0066] See also Figure 1 As shown, the embodiment of the present application provides a data synchronization method, which specifically includes the following steps:

[0067] S101: creating an extensible markup language XML file, where the XML file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between a target table and at least two source tables.

[0068] In the embodiment of the present application, the configuration of the XML file is mainly reflected in:

[0069] The dbMapping.table configuration is no longer limited to a single source table, but supports pointing to at least two source tables, the table names of which are separated by commas, so that multiple source tables can be defined in a single mapping configuration, realizing the expansion of table mapping relationships. At the same time, the dbMapping.targetPk configuration has also been expanded to support primary keys of different source tables, which are also separated by commas, to define the logical connection or correspondence between the target table and at least two source tables, that is, the table mapping relationship.

[0070] In addition, the dbMapping.targetColumns configuration also realizes the transition from single field mapping to multi-source table multi-field mapping. Through comma-separated fields, you can accurately specify which fields in each source table should be mapped to which field in the target table, thereby establishing the field mapping relationship between the target table and the source table.

[0071] In some implementations of the present application, the source table is further divided into a main table and an auxiliary table, that is, the source table in the present application includes the following types: a main table and an auxiliary table. Among them, the main table determines the existence of data, and the auxiliary table is auxiliary information of the main table.

[0072] For example, the target table is the customer information table, and the corresponding source tables are the customer basic information table, customer address information table, customer work information table, etc. In actual business, the customer basic information table data must exist first, and then the customer address information table and customer work information table data. At this time, the customer basic information table is the main table, and the customer address information table and customer work information table are auxiliary tables.

[0073] Assume that the source tables are s1(id, name), s2(id, address), and s3(id, company), where s1 is the primary table, s2 and s3 are auxiliary tables, and the target table is t4(id, name, address, company). Now we need to map s1, s2, and s3 to t4. The configuration is as follows:

[0074] dataSourceKey:sd

[0075] dbMapping:

[0076] database:td

[0077] table:s1,s2,s3

[0078] targetTable:td.t4

[0079] targetPk:

[0080] id:s1.id,s2.id,s3.id

[0081] targetColumns:

[0082] id:s1.id,s2.id,s3.id

[0083] name:s1.name

[0084] address:s2.address

[0085] company:s3.company

[0086] It should be noted that in dbMapping.table, the primary table is placed first and the auxiliary table is placed after the primary table.

[0087] In some implementations of the embodiments of the present application, the XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of the source table, and mapping of target table fields to all fields of the source table.

[0088] Specifically, the source database source and the target database source are used to clarify the data source, and specify which database the source and target of the data synchronization belong to. The source database table and the target database table specifically determine the table range of synchronization, and specify which table is synchronized to which table. The mapping of the target table field and the primary key field of the source table is used to logically define the table mapping relationship between the source table and the target table, and the mapping of the target table field and all the fields of the source table is used to define the field mapping relationship between the source table and the target table. For example, field a1 of source table A corresponds to field b1 of target table B, and field a2 of source table A corresponds to field b2 of target table B.

[0089] S102: configuring the XML file into the canal component, so that the canal component performs corresponding addition, deletion and modification operations on the target table based on the addition, deletion and modification operations of the source table.

[0090] In an embodiment of the present application, the created XML file is configured to the canal-adapter module of the canal component adapter part, and the canal component is provided with the configuration parameters required to perform data synchronization tasks, so that the canal component can respond to data change events from multiple source tables and automatically synchronize and update the target table based on the change operations of multiple source tables. That is to say, according to the new INSERT, DELETE, and UPDATE operations on the source table, the corresponding addition, deletion, and modification operations are automatically performed on the target table.

[0091] In some implementations of the embodiments of the present application, corresponding addition, deletion and modification operations are performed on the target table based on the addition, deletion and modification operations of the source table, including: assembling corresponding structured query statements based on the addition, deletion and modification operations of the source table, and executing the structured query statements in the database to which the target table belongs.

[0092] Specifically, the canal component captures data change operations from the binary log binlog to determine whether they are addition, deletion, or modification operations; then, it matches the primary key and mapping fields according to dbMapping.targetPk and dbMapping.targetColumns in the XML file, assembles the corresponding structured query statements based on the matching primary key and mapping fields, and executes them in the target database to which the target table belongs, so as to automatically synchronize and update the target table based on the change operations of the source table.

[0093] exist Figure 1 In the process shown, first, an extensible markup language XML file is created. The XML1 file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between a target table and at least two source tables; then, the XML file is configured in the canal component, giving the Canal component the ability to map one target table to multiple source tables, thereby enabling the Canal component to directly perform corresponding add, delete, and modify operations on the target table according to the add, delete, and modify operations that occur in the source table. It can be seen that this technical solution effectively avoids the problem of synchronizing multiple table data to the target system separately through one-to-one mapping, and then performing multi-table association queries in the target database, and the problem that the synchronization time increases with the increase of data volume, and has a higher data synchronization rate and reduces data synchronization latency.

[0094] See also Figure 2 As shown, the embodiment of the present application further provides a data synchronization method, which specifically includes the following steps:

[0095] S201: Create an extensible markup language XML file, the XML1 file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between a target table and at least two source tables; the source tables include the following types: primary table and auxiliary table.

[0096] S202: configuring the XML file into the canal component, so that the canal component performs corresponding addition, deletion and modification operations on the target table based on the addition, deletion and modification operations of the source table.

[0097] S203: In response to the deletion of the field in the source table, determine the type of the source table.

[0098] S204: When the source table is the main table, according to the mapping relationship between the target table fields and all the source table fields, the mapping fields corresponding to the deleted fields of the main table are determined in the target table, and the mapping fields corresponding to the deleted fields of the main table are deleted.

[0099] S205: When the source table is an auxiliary table, according to the mapping relationship between the target table field and all the fields of the source table, the mapping field corresponding to the deleted field of the auxiliary table is determined in the target table, and the value of the mapping field corresponding to the deleted field of the auxiliary table is modified to a null value.

[0100] In an embodiment of the present application, first, the data change operation is captured from the binary log binlog to determine whether it is an add, delete or modify operation; then, when it is determined to be a delete operation, the primary key and mapping field are matched through dbMapping.targetPk and dbMapping.targetColumns to determine whether the data change occurs in the main table or the auxiliary table. For the main table, the delete statement is inherited and assembled, and for the auxiliary table, it is rewritten into a modification update statement, and the field matched by dbMapping.targetColumns is mapped to a null value.

[0101] For example, when the source table is an auxiliary table, the statement executed in the database to which the target table belongs is, for example, update t4set t4.company=null; when the source table is a primary table, the statement executed in the database to which the target table belongs is, for example, delete from t4 where t4.id=s1.id.

[0102] It can be seen that when the main table is deleted, the target table is deleted, and when the auxiliary table is deleted, only the auxiliary fields of the target table are deleted - it is actually an update, and the deletion and insertion of the main table are actual deletions and insertions. The deletion and insertion of the auxiliary table are actual updates.

[0103] S206: In response to the insertion of the field in the source table, determining the type of the source table;

[0104] S207: When the source table is the primary table, according to the mapping relationship between the target table field and all the source table fields, a mapping field corresponding to the field inserted into the primary table is determined in the target table, and an insertion value of the field inserted into the primary table is inserted into the mapping field corresponding to the field inserted into the primary table;

[0105] S208: When the source table is an auxiliary table, according to the mapping relationship between the target table field and all the fields of the source table, the mapping field corresponding to the field inserted into the auxiliary table is determined in the target table, and the value of the mapping field corresponding to the field inserted into the auxiliary table is modified to the insertion value of the field inserted into the auxiliary table.

[0106] In an embodiment of the present application, first, the data change operation is captured from the binary log binlog to determine whether it is an add, delete or modify operation; then, when it is determined to be an insert operation, the primary key and mapping field are matched through dbMapping.targetPk and dbMapping.targetColumns to determine whether the data change occurs in the main table or the auxiliary table. For the main table, an insert statement is inherited and assembled to insert the value of the field where the data change occurs in the main table into the field matched according to dbMapping.targetColumns; and for the auxiliary table, it is rewritten into a modified update statement to modify the field matched by dbMapping.targetColumns to the value inserted into the field where the data change occurs in the main table.

[0107] For example, when the source table is an auxiliary table, the statement executed in the database to which the target table belongs is: update t4set t4.address=s2.address where s4.id=s2.id; when the source table is a primary table, the statement executed in the database to which the target table belongs is: insert into t4(id,name)values(s1.id,s1.name).

[0108] It can be seen that when the main table is inserted, the target table also performs an insert operation, and when the auxiliary table is inserted, the target table performs a modification operation. The insert of the main table is an actual insert, and the insert of the auxiliary table is actually an update.

[0109] S209: In response to the modification of the fields in the source table, according to the mapping relationship between the target table fields and all the fields in the source table, the mapping fields corresponding to the modified fields in the source table are determined in the target table, and the values ​​of the mapping fields corresponding to the modified fields in the auxiliary table are modified to the values ​​of the modified fields in the source table.

[0110] In an embodiment of the present application, after determining that the data change operation captured in the binary log binlog is a modification, the primary key and mapping field corresponding to the modified field of the source table in the target table are matched according to the source table to dbMapping.targetPk and dbMapping.targetColumns; then, according to the matched primary key and mapping field, the modification update statement is inherited and assembled, and the update statement is executed in the target database to update the modified value of the source table to the mapping field in the target table accordingly. For example, the statement executed in the database to which the target table belongs is update t4 sett4.company=s3.company where t4.id=s3.id.

[0111] The above steps can also be done by Figure 3 Specifically, Figure 3 It can be seen intuitively that the canal component performs corresponding operations on the target table according to the operation type. If the operation type on the source table is insert, it is necessary to further determine whether the source table is the main table or the auxiliary table. If the source table is the main table, the insert operation is performed on the target table; if the source table is the auxiliary table, the modification operation is performed on the target table. If the operation type on the source table is modify, the assembly insert operation on the target table is inherited. If the operation type on the source table is delete, it is necessary to further determine whether the source table is the main table or the auxiliary table. If the source table is the main table, the delete operation is performed on the target table; if the source table is the auxiliary table, the modification operation is performed on the target table.

[0112] exist Figure 2 In the process shown, when the data in the source table changes, the target table can synchronously reflect these changes, which helps maintain the consistency of data between different data tables and avoids data redundancy and data inconsistency. In particular, for deletion and insertion operations, by distinguishing between the main table and the auxiliary table, different processing methods are performed. When the main table is deleted, the target table is actually deleted, and when the auxiliary table is deleted, only the auxiliary fields of the target table are deleted. When the main table is inserted, the target table is actually inserted, and when the auxiliary table is inserted, the target table is modified, so that data synchronization is more in line with actual business needs.

[0113] See also Figure 4 As shown, an embodiment of the present application provides a data synchronization device, the device comprising:

[0114] A file creation module 401 is used to create an extensible markup language XML file, wherein the XML1 file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between a target table and at least two source tables;

[0115] The file configuration module 402 is used to configure the XML file into the canal component so that the canal component performs corresponding addition, deletion and modification operations on the target table based on the addition, deletion and modification operations of the source table.

[0116] In some implementations of the embodiments of the present application, the source table includes the following types: a main table and an auxiliary table.

[0117] In some implementations of the embodiments of the present application, the device includes: a deletion module, the deletion module is used to:

[0118] In response to the deletion of a field in the source table, determining the type of the source table;

[0119] When the source table is the primary table, based on the mapping relationship between the target table fields and all the source table fields, the mapping fields corresponding to the deleted fields of the primary table are determined in the target table, and the mapping fields corresponding to the deleted fields of the primary table are deleted.

[0120] When the source table is an auxiliary table, according to the mapping relationship between the target table fields and all the fields of the source table, the mapping fields corresponding to the fields deleted from the auxiliary table are determined in the target table, and the values ​​of the mapping fields corresponding to the fields deleted from the auxiliary table are modified to null values.

[0121] In some implementations of the embodiments of the present application, the device further includes: an insertion module, the insertion module being configured to:

[0122] In response to the insertion of a field in a source table, determining the type of the source table;

[0123] When the source table is the primary table, according to the mapping relationship between the target table field and all the fields of the source table, the mapping field corresponding to the field inserted in the primary table is determined in the target table, and the insertion value of the field inserted in the primary table is inserted in the mapping field corresponding to the field inserted in the primary table;

[0124] When the source table is an auxiliary table, based on the mapping relationship between the target table fields and all the fields of the source table, the mapping fields corresponding to the fields inserted into the auxiliary table are determined in the target table, and the values ​​of the mapping fields corresponding to the fields inserted into the auxiliary table are modified to the insertion values ​​of the fields inserted into the auxiliary table.

[0125] In some implementations of the embodiments of the present application, the device further includes: a modification module, the modification module being configured to:

[0126] In response to the modification of the fields in the source table, based on the mapping relationship between the target table fields and all the fields in the source table, the mapping fields corresponding to the modified fields in the source table are determined in the target table, and the values ​​of the mapping fields corresponding to the modified fields in the auxiliary table are modified to the values ​​of the modified fields in the source table.

[0127] In some implementations of the embodiments of the present application, the XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of the source table, and mapping of target table fields to all fields of the source table.

[0128] In some implementations of the embodiments of the present application, the target table is subjected to corresponding addition, deletion, and modification operations based on the addition, deletion, and modification operations of the source table, including:

[0129] Based on the addition, deletion and modification operations of the source table, assemble the corresponding structured query statements and execute the structured query statements in the database to which the target table belongs.

[0130] A third aspect of the present application provides a data synchronization device, which includes a memory and a processor, and the processor is used to execute a program stored in the memory and run any data synchronization method provided in the first aspect of the present application.

[0131] The fourth aspect of the present application provides a readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, each step of any data synchronization method provided in the first aspect of the present application is implemented.

[0132] like Figure 5 As shown, the embodiment of the present application further provides a device, including: a memory 501, a processor 502;

[0133] Wherein, the memory 501 is used to store programs;

[0134] The processor 502 is used to execute the program in the memory to implement each step of the data synchronization method provided in the embodiment of the present application.

[0135] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned data synchronization method.

[0136] Finally, it should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0137] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization method, characterized in that: The method comprises: Creating an extensible markup language XML file, wherein the XML file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between the target table and at least two source tables; The XML file is configured in the canal component so that the canal component performs corresponding addition, deletion and modification operations on the target table based on the addition, deletion and modification operations of the source table.

2. The method according to claim 1, characterized in that The source tables include the following types: primary tables and auxiliary tables.

3. The method according to claim 2, characterized in that The method comprises: In response to the deletion of the field in the source table, determining the type of the source table; When the source table is the main table, according to the mapping relationship between the target table field and all the fields of the source table, the mapping field corresponding to the deleted field of the main table is determined in the target table, and the mapping field corresponding to the deleted field of the main table is deleted; When the source table is the auxiliary table, according to the mapping relationship between the target table fields and all the fields of the source table, the mapping fields corresponding to the fields deleted from the auxiliary table are determined in the target table, and the values ​​of the mapping fields corresponding to the fields deleted from the auxiliary table are modified to null values.

4. The method according to claim 2, characterized in that: The method further comprises: In response to the insertion of a field in the source table, determining the type of the source table; When the source table is the main table, according to the mapping relationship between the target table field and all the fields of the source table, a mapping field corresponding to the field inserted into the main table is determined in the target table, and the insertion value of the field inserted into the main table is inserted into the mapping field corresponding to the field inserted into the main table; When the source table is an auxiliary table, based on the mapping relationship between the target table fields and all the fields of the source table, the mapping fields corresponding to the fields inserted into the auxiliary table are determined in the target table, and the values ​​of the mapping fields corresponding to the fields inserted into the auxiliary table are modified to the insertion values ​​of the fields inserted into the auxiliary table.

5. The method according to claim 1, characterized in that: The method further comprises: In response to the modification of the fields in the source table, based on the mapping relationship between the target table fields and all the fields in the source table, the mapping fields corresponding to the modified fields in the source table are determined in the target table, and the values ​​of the mapping fields corresponding to the modified fields in the auxiliary table are modified to the values ​​of the modified fields in the source table.

6. The method according to claim 1, characterized in that The XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of source table, and mapping of target table fields to all fields of source table.

7. The method according to claim 1, characterized in that The adding, deleting and modifying operations on the target table based on the adding, deleting and modifying operations on the source table include: Based on the addition, deletion and modification operations of the source table, a corresponding structured query statement is assembled, and the structured query statement is executed in the database to which the target table belongs.

8. A data synchronization device, characterized in that: The device comprises: A file creation module, used to create an extensible markup language XML file, wherein the XML file is used to establish a table mapping relationship between a target table and at least two source tables, and to establish a field mapping relationship between the target table and at least two source tables; The file configuration module is used to configure the XML file into the canal component so that the canal component performs corresponding addition, deletion and modification operations on the target table based on the addition, deletion and modification operations of the source table.

9. A data synchronization device, characterized in that: The device includes a memory and a processor, and the processor is used to execute the program stored in the memory to run the data synchronization method as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the data synchronization method according to any one of claims 1 to 7 is implemented.

Images