A data synchronization method, apparatus, device, and readable storage medium

Abstract

This application discloses a data synchronization method, apparatus, device, and readable storage medium, relating to the field of big data. The method includes: creating an Extensible Markup Language (XML) file, establishing a table mapping relationship between a target table and at least two source tables through the XML file, and establishing a field mapping relationship between the target table and at least two source tables; then, configuring the XML file into a Canal component so that the Canal component performs corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations of the source tables, thereby improving the data synchronization rate.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data synchronization method, apparatus, device, and readable storage medium. Background Technology

[0002] Within enterprises, data is typically persistently stored in MySQL databases across various business systems. Given the limitations of MySQL in handling specific application scenarios, it is usually necessary to synchronize MySQL data in real-time to other database systems such as Hive, TiDB, Kudu, HBase, and Elasticsearch to meet diverse business needs.

[0003] Alibaba's open-source Canal middleware cleverly leverages MySQL's master-slave replication mechanism by posing as a MySQL slave. This mechanism captures the data change logs (binlog) of the MySQL master database and sends these changes to a message queue (MQ) in real time. Subsequently, this changed data is distributed to downstream database systems, achieving real-time data synchronization. This data synchronization process can take milliseconds or even sub-seconds.

[0004] However, the Canal component currently does not support many-to-one data mapping, meaning it cannot directly map data from multiple source tables to a single target table. This is particularly inconvenient when converting Online Transaction Processing (OLTP) tables to Online Analytical Processing (OLAP) tables, as OLAP tables are often designed to merge multiple related narrow tables into a single wide table.

[0005] Currently, one feasible solution to this problem is to first perform a one-to-one mapping between multiple tables, and then perform multi-table join queries in the target database system. However, this method has significant performance bottlenecks, especially in real-time data synchronization scenarios. Since business operations typically tolerate data latency in the millisecond range, multi-table join queries can consume several seconds or even minutes, which is clearly unacceptable. For example, suppose tables A, B, and C each have 1 million records, and each table experiences approximately 10 changes per second daily. Although Canal can synchronize this changed data to the target system at sub-second speeds, subsequent join queries on these three tables with millions of records in the target system could take tens of seconds, severely violating real-time requirements.

[0006] Therefore, improving the data synchronization rate under many-to-one mapping has become an urgent technical problem to be solved. Summary of the Invention

[0007] In view of this, the main objective of this application is to provide a time synchronization method, apparatus, device and readable storage medium, with the aim of improving the data synchronization rate.

[0008] The first aspect of this application provides a data synchronization method, the method comprising:

[0009] Create an Extensible Markup Language (XML) file. The XML file is used to establish the table mapping relationship between the target table and at least two source tables, as well as the field mapping relationship between the target table and at least two source tables.

[0010] Configure the XML file into the Canal component so that the Canal component can perform corresponding CRUD operations on the target table based on the CRUD operations of the source table.

[0011] In some implementations of the first aspect of this application, the source table includes the following types: main table and auxiliary table.

[0012] In some implementations of the first aspect of this application, the method includes:

[0013] In response to the deletion of a field in the source table, determine the type of the source table;

[0014] When the source table is the main table, based on the mapping relationship between the fields in the target table and all fields in the source table, determine the mapping field in the target table corresponding to the field deleted in the main table, and then delete the mapping field corresponding to the field deleted in the main table.

[0015] When the source table is an auxiliary table, based on the mapping relationship between the fields in the target table and all fields in the source table, determine the mapping field in the target table corresponding to the field deleted in the auxiliary table, and modify the value of the mapping field corresponding to the field deleted in the auxiliary table to a null value.

[0016] In some implementations of the first aspect of this application, the method further includes:

[0017] In response to the insertion of a field in the source table, determine the type of the source table;

[0018] When the source table is the main table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table that corresponds to the field inserted in the main table, and insert the value of the field inserted in the main table into the mapping field that corresponds to the field inserted in the main table.

[0019] When the source table is an auxiliary table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table corresponding to the field inserted in the auxiliary table, and modify the value of the mapping field corresponding to the field inserted in the auxiliary table to the inserted value of the field inserted in the auxiliary table.

[0020] In some implementations of the first aspect of this application, the method further includes:

[0021] In response to a modification to a field in the source table, based on the mapping relationship between the target table fields and all fields in the source table, a mapping field corresponding to the modified field in the source table is determined in the target table, and the value of the mapping field corresponding to the modified field in the auxiliary table is modified to the value of the modified field in the source table.

[0022] In some implementations of the first aspect of this application, the XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of the source table, and mapping of target table fields to all fields of the source table.

[0023] In some implementations of the first aspect of this application, corresponding add, delete, and modify operations are performed on the target table based on the add, delete, and modify operations of the source table, including:

[0024] Based on the CRUD operations on the source table, assemble the corresponding structured query statements and execute the structured query statements in the database to which the target table belongs.

[0025] A second aspect of this application provides a data synchronization apparatus, the apparatus comprising:

[0026] The file creation module is used to create Extensible Markup Language (XML) files. The XML files are used to establish table mapping relationships between the target table and at least two source tables, as well as field mapping relationships between the target table and at least two source tables.

[0027] The file configuration module is used to configure XML files into the Canal component, so that the Canal component can perform corresponding CRUD operations on the target table based on the CRUD operations of the source table.

[0028] A third aspect of this application provides a data synchronization device, which includes a memory and a processor, the processor being configured to execute a program stored in the memory and run any of the data synchronization methods provided in the first aspect of this application.

[0029] The fourth aspect of this application provides a readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the data synchronization methods provided in the first aspect of this application.

[0030] The technical solution provided in this application has the following beneficial effects:

[0031] In the technical solution provided in this application, firstly, an Extensible Markup Language (XML) file is created. This XML file is used to establish the table mapping relationship between the target table and at least two source tables, as well as the field mapping relationship between the target table and at least two source tables. Then, the XML file is configured into the Canal component, giving the Canal component the ability to map one target table to multiple source tables. This allows the Canal component to directly perform corresponding add, delete, and update operations on the target table based on the add, delete, and update operations that occur in the multiple source tables. Therefore, this technical solution effectively avoids the problem that the synchronization time increases with the amount of data when synchronizing data from multiple tables to the target system through a one-to-one mapping and then performing multi-table join queries in the target database. It achieves a higher data synchronization rate and reduces synchronization latency. Attached Figure Description

[0032] Figure 1 A flowchart illustrating a data synchronization method provided in an embodiment of this application;

[0033] Figure 2 A flowchart illustrating yet another data synchronization method provided in an embodiment of this application;

[0034] Figure 3 A flowchart illustrating the process by which the canal component provided in this application performs corresponding operations on the target table based on the operation type of the source table;

[0035] Figure 4 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] In the Canal component, single-table data mapping can be achieved by configuring an XML file in the Canal-Adapter module. For example:

[0039] #Data source;

[0040] dataSourceKey:defaultDS

[0041] #Database mapping configuration section;

[0042] dbMapping:

[0043] #Specify the name of the source database;

[0044] database:mytest

[0045] # Specifies the table name in the source database;

[0046] table:user

[0047] #Specify the table name in the target database;

[0048] targetTable:mytest2.user

[0049] # Primary key configuration for the target table

[0050] targetPk:

[0051] id:id

[0052] #Target field mapping configuration

[0053] targetColumns:

[0054] id:id

[0055] name:name

[0056] Here, dataSourceKey and dbMapping.database represent the source database and target database, respectively; dbMapping.table and dbMapping.targetTable represent the source database table and target database table, respectively; dbMapping.targetPk represents the mapping between target table fields and primary key fields of the source table; and dbMapping.targetColumns represents the mapping between target table fields and all fields of the source table.

[0057] It should be noted that the above is an implementation of one-to-one mapping. For one-to-many mapping, that is, mapping one source table to multiple target tables, multiple XML files can be configured. Each XML file indicates the same source database table, but the target database tables and field mappings are different.

[0058] The canal component can parse binary logs (binlog), combining the insert, delete, and update operations from the source table with the mapping file to perform insert, delete, and update operations on the target table and fields.

[0059] For example, source table S has fields T1, T2, and T3, and target table T has corresponding fields T11, T22, and T33. The structured query SQL statement for an insert operation on the source table is:

[0060] insert S(T1,T2,T3)values(V1,V2,V3);

[0061] Therefore, the SQL statement for the Canal component to perform operations on the target table is:

[0062] insert T(T11,T22,T33)values(V1,V2,V3);

[0063] Understandably, the deletion and modification operations are the same as above.

[0064] Currently, the Canal component only supports one-to-one mapping and does not provide a many-to-one mapping implementation, meaning multiple source tables can be mapped to a single target table. In practice, when converting OLTP tables to OLAP tables, it's necessary to map multiple narrow tables to a single wide table. Therefore, OLTP table design aims to atomically split multiple tables as much as possible. OLAP table design, on the other hand, requires grouping related data into a single table whenever possible. This inherent design difference leads to numerous many-to-one data synchronization mapping requirements. Under existing mechanisms, a feasible approach is to first perform one-to-one mappings across multiple tables and then perform multi-table joins in the target database system. However, this approach has a significant drawback: low performance in multi-table joins. Especially when Canal is used in real-time data synchronization scenarios, where the acceptable latency for the business is in the millisecond range, joins involving multiple tables, particularly large ones, can result in join calculations taking seconds or even minutes, which is unacceptable to the business.

[0065] In view of this, embodiments of this application provide a data synchronization method, apparatus, device, and readable storage medium, with the aim of improving the data synchronization rate and reducing the data synchronization latency.

[0066] See Figure 1 As shown in the figure, this application provides a data synchronization method, which specifically includes the following steps:

[0067] S101: Create an Extensible Markup Language (XML) file. The XML file is used to establish table mapping relationships between the target table and at least two source tables, as well as field mapping relationships between the target table and at least two source tables.

[0068] In the embodiments of this application, the configuration specificity of the XML file is mainly reflected in:

[0069] The `dbMapping.table` configuration is no longer limited to a single source table, but supports pointing to at least two source tables. These source table names are separated by commas, allowing multiple source tables to be defined within a single mapping configuration, thus expanding the table mapping relationship. Simultaneously, the `dbMapping.targetPk` configuration has also been extended to support primary keys from different source tables. These primary keys are also separated by commas to define the logical connection or correspondence between the target table and at least two source tables, i.e., the table mapping relationship.

[0070] Furthermore, the dbMapping.targetColumns configuration also enables the transformation from single-field mapping to multi-field mapping across multiple source tables. By using comma-separated fields, you can precisely specify which fields in each source table should be mapped to which fields in the target table, thus establishing the field mapping relationship between the target and source tables.

[0071] In some implementations of this application, the source table is further divided into a main table and an auxiliary table. That is, the source table in this application includes the following types: a main table and an auxiliary table. The main table determines the existence of data, while the auxiliary table provides supplementary information to the main table.

[0072] For example, the target table is the customer information table, corresponding to the source tables such as the customer basic information table, customer address information table, and customer employment information table. In actual business operations, the customer basic information table must exist first, followed by the customer address information table and the customer employment information table. In this case, the customer basic information table is the main table, while the customer address information table and the customer employment information table are auxiliary tables.

[0073] Assuming the source tables are s1(id, name), s2(id, address), and s3(id, company), where s1 is the main table and s2 and s3 are auxiliary tables, and the target table is t4(id, name, address, company), the goal is to map s1, s2, and s3 to table t4. The configuration would be as follows:

[0074] dataSourceKey:sd

[0075] dbMapping:

[0076] database:td

[0077] table:s1,s2,s3

[0078] targetTable:td.t4

[0079] targetPk:

[0080] id:s1.id,s2.id,s3.id

[0081] targetColumns:

[0082] id:s1.id,s2.id,s3.id

[0083] name:s1.name

[0084] address:s2.address

[0085] company:s3.company

[0086] It should be noted that in dbMapping.table, the main table is placed first, and the auxiliary tables are placed after the main table.

[0087] In some implementations of this application, the XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of source table, and mapping of target table fields to all fields of source table.

[0088] Specifically, the source database and target database are used to clarify the data source, specifying which database the data synchronization originates from and targets. The source database table and target database table define the scope of tables to be synchronized, specifying which table's synchronization will extend to which table. The mapping between target table fields and the primary key fields of the source table logically defines the table mapping relationship between the source and target tables. The mapping between target table fields and all fields of the source table defines the field mapping relationship between the source and target tables; for example, field a1 in source table A corresponds to field b1 in target table B, and field a2 in source table A corresponds to field b2 in target table B.

[0089] S102: Configure the XML file into the canal component so that the canal component can perform corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations of the source table.

[0090] In the embodiments of this application, the created XML file is configured in the canal-adapter module of the canal component adapter section, providing the canal component with the configuration parameters required to perform data synchronization tasks. This enables the canal component to respond to data change events from multiple source tables and automatically update the target table based on the change operations of multiple source tables. In other words, based on the INSERT, DELETE, and UPDATE operations on the source tables, the corresponding INSERT, DELETE, and UPDATE operations are automatically performed on the target table.

[0091] In some implementations of this application, the corresponding add, delete, and modify operations are performed on the target table based on the add, delete, and modify operations of the source table, including: assembling the corresponding structured query statement based on the add, delete, and modify operations of the source table, and executing the structured query statement in the database to which the target table belongs.

[0092] Specifically, the Canal component captures data change operations from the binary log (binlog) to determine whether they are additions, deletions, or modifications. Then, it matches the primary key and mapping fields based on dbMapping.targetPk and dbMapping.targetColumns in the XML file. Based on the matched primary key and mapping fields, it assembles the corresponding structured query statement and executes it in the target database to which the target table belongs, so as to automatically update the target table synchronously based on the change operations of the source table.

[0093] exist Figure 1 The illustrated process first involves creating an Extensible Markup Language (XML) file. This XML file establishes the table mapping relationship between the target table and at least two source tables, as well as the field mapping relationship between the target table and at least two source tables. Then, the XML file is configured into the Canal component, granting it the ability to map one target table to multiple source tables. This allows the Canal component to directly perform corresponding add, delete, and update operations on the target table based on the add, delete, and update operations occurring in the source tables. Therefore, this technical solution effectively avoids the problem of synchronization time increasing with data volume when synchronizing multiple tables to the target system through one-to-one mapping and then performing multi-table join queries in the target database. It achieves a higher data synchronization rate and reduces data synchronization latency.

[0094] See Figure 2 As shown in the figure, this application embodiment provides another data synchronization method, which specifically includes the following steps:

[0095] S201: Create an Extensible Markup Language (XML) file. The XML file is used to establish the table mapping relationship between the target table and at least two source tables, as well as the field mapping relationship between the target table and at least two source tables. The source tables include the following types: main table and auxiliary table.

[0096] S202: Configure the XML file into the canal component so that the canal component can perform corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations of the source table.

[0097] S203: In response to the deletion of a field in the source table, determine the type of the source table.

[0098] S204: When the source table is the main table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table corresponding to the field deleted in the main table, and delete the mapping field corresponding to the field deleted in the main table.

[0099] S205: When the source table is an auxiliary table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table corresponding to the field deleted in the auxiliary table, and modify the value of the mapping field corresponding to the field deleted in the auxiliary table to a null value.

[0100] In the embodiments of this application, firstly, data change operations are captured from the binary log (binlog) to determine whether they are insert, delete, or update operations. Then, when it is determined to be a delete operation, the primary key and mapping fields are matched using dbMapping.targetPk and dbMapping.targetColumns to determine whether the data change occurred in the main table or the auxiliary table. For the main table, a delete statement is constructed by inheritance, while for the auxiliary table, it is rewritten as an update statement, mapping the fields matched by dbMapping.targetColumns to null values.

[0101] For example, when the source table is an auxiliary table, the statement executed in the database to which the target table belongs is, for example: update t4 set t4.company = null; when the source table is the main table, the statement executed in the database to which the target table belongs is, for example: delete from t4 where t4.id = s1.id.

[0102] Therefore, when deleting from the main table, the target table is also deleted; when deleting from the auxiliary table, only the auxiliary fields of the target table are deleted—essentially an update. Deletions and inserts in the main table are actual deletions and inserts, while deletions and inserts in the auxiliary table are actual updates.

[0103] S206: In response to the insertion of a field in the source table, determine the type of the source table;

[0104] S207: When the source table is the main table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table corresponding to the field inserted in the main table, and insert the inserted value of the field in the main table into the mapping field corresponding to the field inserted in the main table.

[0105] S208: When the source table is an auxiliary table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table corresponding to the field inserted in the auxiliary table, and modify the value of the mapping field corresponding to the field inserted in the auxiliary table to the inserted value of the field inserted in the auxiliary table.

[0106] In the embodiments of this application, firstly, data change operations are captured from the binary log (binlog) to determine whether they are insert, delete, or update operations. Then, when it is determined to be an insert operation, the primary key and mapping fields are matched using dbMapping.targetPk and dbMapping.targetColumns to determine whether the data change occurred in the main table or the auxiliary table. For the main table, an insert statement is constructed by inheritance, inserting the value of the field in the main table that has undergone data change into the field matched by dbMapping.targetColumns. For the auxiliary table, an update statement is rewritten to modify the field matched by dbMapping.targetColumns into the value of the field in the main table that has undergone data change.

[0107] For example, when the source table is an auxiliary table, the statement executed in the database to which the target table belongs is, for example: update t4 set t4.address = s2.address where s4.id = s2.id; when the source table is the main table, the statement executed in the database to which the target table belongs is, for example: insert into t4(id,name)values(s1.id,s1.name).

[0108] Therefore, when inserting into the main table, the target table also performs an insert operation, while when inserting into the auxiliary table, the target table performs a modification operation. Insertion into the main table is an actual insert, while insertion into the auxiliary table is actually an update.

[0109] S209: In response to a modification of a field in the source table, based on the mapping relationship between the target table field and all fields in the source table, determine the mapping field in the target table corresponding to the modified field in the source table, and modify the value of the mapping field corresponding to the modified field in the auxiliary table to the value of the modified field in the source table.

[0110] In the embodiments of this application, after determining that the data change operation captured in the binary log (binlog) is a modification, the primary key and mapping fields in the target table corresponding to the modified fields in the source table are matched according to the source table's `dbMapping.targetPk` and `dbMapping.targetColumns`. Then, based on the matched primary key and mapping fields, a modification update statement is assembled and executed in the target database to update the modified values ​​in the source table to the corresponding mapping fields in the target table. For example, the statement executed in the database to which the target table belongs is `update t4 sett4.company=s3.company where t4.id=s3.id`.

[0111] The above steps can also be achieved through Figure 3 To explain. Specifically, by Figure 3 It's clear that the Canal component performs corresponding operations on the target table based on the operation type. If the operation type on the source table is insert, it needs to further determine whether the source table is a primary table or a secondary table. If the source table is a primary table, an insert operation is performed on the target table; if the source table is a secondary table, an update operation is performed on the target table. If the operation type on the source table is update, it inherits the assembled insert operation from the target table. If the operation type on the source table is delete, it needs to further determine whether the source table is a primary table or a secondary table. If the source table is a primary table, a delete operation is performed on the target table; if the source table is a secondary table, an update operation is performed on the target table.

[0112] exist Figure 2 In the illustrated process, when data in the source table changes, the target table can synchronously reflect these changes, helping to maintain data consistency across different tables and avoiding data redundancy and inconsistency. Specifically, for delete and insert operations, different processing methods are applied by distinguishing between the main table and auxiliary tables. When deleting from the main table, the target table performs the actual deletion; when deleting from the auxiliary table, only the auxiliary fields in the target table are deleted. When inserting from the main table, the target table performs the actual insertion; when inserting from the auxiliary table, the target table is modified. This makes data synchronization more aligned with actual business needs.

[0113] See Figure 4 As shown in the figure, this application provides a data synchronization device, which includes:

[0114] The file creation module 401 is used to create an Extensible Markup Language (XML) file. The XML file is used to establish the table mapping relationship between the target table and at least two source tables, and to establish the field mapping relationship between the target table and at least two source tables.

[0115] The file configuration module 402 is used to configure the XML file into the canal component so that the canal component can perform corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations of the source table.

[0116] In some implementations of this application, the source table includes the following types: main table and auxiliary table.

[0117] In some implementations of this application, the apparatus includes a deletion module, which is used to:

[0118] In response to the deletion of a field in the source table, determine the type of the source table;

[0119] When the source table is the main table, based on the mapping relationship between the fields in the target table and all fields in the source table, determine the mapping field in the target table corresponding to the field deleted in the main table, and then delete the mapping field corresponding to the field deleted in the main table.

[0120] When the source table is an auxiliary table, based on the mapping relationship between the fields in the target table and all fields in the source table, determine the mapping field in the target table corresponding to the field deleted in the auxiliary table, and modify the value of the mapping field corresponding to the field deleted in the auxiliary table to a null value.

[0121] In some implementations of the embodiments of this application, the apparatus further includes: an insertion module, the insertion module being used for:

[0122] In response to the insertion of a field in the source table, determine the type of the source table;

[0123] When the source table is the main table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table that corresponds to the field inserted in the main table, and insert the value of the field inserted in the main table into the mapping field that corresponds to the field inserted in the main table.

[0124] When the source table is an auxiliary table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table corresponding to the field inserted in the auxiliary table, and modify the value of the mapping field corresponding to the field inserted in the auxiliary table to the inserted value of the field inserted in the auxiliary table.

[0125] In some implementations of the embodiments of this application, the apparatus further includes: a modification module, which is used to:

[0126] In response to a modification to a field in the source table, based on the mapping relationship between the target table fields and all fields in the source table, a mapping field corresponding to the modified field in the source table is determined in the target table, and the value of the mapping field corresponding to the modified field in the auxiliary table is modified to the value of the modified field in the source table.

[0127] In some implementations of this application, the XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of source table, and mapping of target table fields to all fields of source table.

[0128] In some implementations of this application, corresponding add, delete, and modify operations are performed on the target table based on the add, delete, and modify operations of the source table, including:

[0129] Based on the CRUD operations on the source table, assemble the corresponding structured query statements and execute the structured query statements in the database to which the target table belongs.

[0130] A third aspect of this application provides a data synchronization device, which includes a memory and a processor, the processor being configured to execute a program stored in the memory and run any of the data synchronization methods provided in the first aspect of this application.

[0131] The fourth aspect of this application provides a readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the data synchronization methods provided in the first aspect of this application.

[0132] like Figure 5 As shown in the figure, this application embodiment also provides a device, including: a memory 501 and a processor 502;

[0133] Among them, memory 501 is used to store programs;

[0134] The processor 502 is used to execute a program in memory to implement the various steps of the data synchronization method provided in the embodiments of this application.

[0135] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the data synchronization method.

[0136] Finally, it should be noted that in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization method, characterized in that, The method includes: Create an Extensible Markup Language (XML) file, which is used to establish table mapping relationships between the target table and at least two source tables, and to establish field mapping relationships between the target table and at least two source tables; Configure the XML file into the canal component so that the canal component can perform corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations of the source table.

2. The method according to claim 1, characterized in that, The source tables include the following types: main table and auxiliary table.

3. The method according to claim 2, characterized in that, The method includes: In response to the deletion of a field in the source table, determine the type of the source table; When the source table is the main table, based on the mapping relationship between the fields of the target table and all fields of the source table, determine the mapping field in the target table corresponding to the deleted field in the main table, and delete the mapping field corresponding to the deleted field in the main table; When the source table is the auxiliary table, based on the mapping relationship between the fields of the target table and all fields of the source table, a mapping field corresponding to the deleted field in the auxiliary table is determined in the target table, and the value of the mapping field corresponding to the deleted field in the auxiliary table is modified to a null value.

4. The method according to claim 2, characterized in that, The method further includes: In response to the insertion of a field in the source table, determine the type of the source table; When the source table is the main table, according to the mapping relationship between the fields of the target table and all fields of the source table, the mapping field corresponding to the field inserted in the main table is determined in the target table, and the inserted value of the field inserted in the main table is inserted in the mapping field corresponding to the field inserted in the main table. When the source table is an auxiliary table, based on the mapping relationship between the fields of the target table and all fields of the source table, a mapping field corresponding to the field inserted in the auxiliary table is determined in the target table, and the value of the mapping field corresponding to the field inserted in the auxiliary table is modified to the insertion value of the field inserted in the auxiliary table.

5. The method according to claim 2, characterized in that, The method further includes: In response to a modification of a field in the source table, based on the mapping relationship between the target table fields and all fields in the source table, a mapping field corresponding to the modified field in the source table is determined in the target table, and the value of the mapping field corresponding to the modified field in the auxiliary table is modified to the value of the modified field in the source table.

6. The method according to claim 1, characterized in that, The XML file includes the following variables: source database source, target database source, source database table, target database table, mapping of target table fields to primary key fields of the source table, and mapping of target table fields to all fields of the source table.

7. The method according to claim 1, characterized in that, The corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations of the source table include: Based on the add, delete, and modify operations of the source table, assemble the corresponding structured query statement, and execute the structured query statement in the database to which the target table belongs.

8. A data synchronization device, characterized in that, The device includes: The file creation module is used to create Extensible Markup Language (XML) files, which are used to establish table mapping relationships between the target table and at least two source tables, as well as field mapping relationships between the target table and at least two source tables. The file configuration module is used to configure the XML file into the canal component, so that the canal component can perform corresponding add, delete, and modify operations on the target table based on the add, delete, and modify operations of the source table.

9. A data synchronization device, characterized in that, The device includes a memory and a processor, the processor being configured to execute a program stored in the memory, running the data synchronization method as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the various steps of the data synchronization method as described in any one of claims 1 to 7.

Images