DDL operation synchronization method, device and electronic device

Abstract

The embodiments of the present invention provide a DDL operation synchronization method, device and electronic device, which relate to the field of data migration technology. The method includes: traversing each database instance of a distributed database, and when traversing each database instance, obtaining a first log record for a target subtable in the database instance; detecting whether there is a target record matching the first log record in a log file for storing a second log record of a target data table; if so, traversing the next database instance; if not, adding a target record to the log file, performing a target DDL operation on the target data table, and traversing the next database instance. Compared with the prior art, the solution provided by the embodiments of the present invention can solve the problem of repeated DDL operations caused by the same DDL operations of multiple subtables belonging to the same data table in a distributed database being synchronized to the data table in a centralized database.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a DDL operation synchronization method, device and electronic equipment. Background Art

[0002] In the traditional data migration process, it usually involves bidirectional migration of distributed databases and centralized databases. After migrating each sub-table stored in each database instance of the distributed database to the centralized database, each database instance can still store each sub-table. In addition, users can perform DDL (Data Definition Language) operations on these sub-tables and synchronize the performed DDL operations to the corresponding data tables in the centralized database.

[0003] The so-called DDL operation refers to an operation that changes the structure of a data table, such as deleting a column in a data table, adding a column in a data table, modifying a column in a data table, deleting a table, etc.

[0004] In the related art, during the synchronization of DDL operations, the same DDL operations of multiple sub-tables belonging to the same data table in the distributed database are synchronized to the data table in the centralized database, which may cause the problem of repeated DDL operations. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a DDL operation synchronization method, device and electronic device to solve the problem of repeated DDL operations caused by the same DDL operations of multiple sub-tables belonging to the same data table in a distributed database being synchronized to the data table in a centralized database. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a DDL operation synchronization method, the method comprising:

[0007] Traversing each database instance of the distributed database, when traversing to each database instance, obtaining a first log record for a target subtable in the database instance; wherein the target subtable is a subtable of a target data table in a centralized database; and the first log record is used to record a target DDL operation performed on the target subtable;

[0008] Detecting whether there is a target record matching the first log record in a log file for storing a second log record of the target data table; wherein the second log record is used to record a DDL operation performed on the target data table; the DDL operation recorded in the target record is the target DDL operation, and the occurrence time information of the recorded DDL operation matches the occurrence time information of the target DDL operation;

[0009] If it exists, traverse the next database instance;

[0010] If it does not exist, the target record is added to the log file, the target DDL operation is performed on the target data table, and the next database instance is traversed.

[0011] Optionally, in a specific implementation, the first log record includes operation information of a target DDL operation performed on the target sub-table, and a global timestamp corresponding to the target DDL operation;

[0012] The second log record includes operation information of the DDL operation performed on the target data table and a global timestamp corresponding to the performed DDL operation;

[0013] The global timestamp corresponding to any DDL operation is: the timestamp set for the DDL operation by the computing node of the distributed database after receiving the operation instruction of the DDL operation;

[0014] The detecting whether there is a target record matching the first log record in a log file for storing the second log record of the target data table includes:

[0015] It is detected whether there is a target record including operation information of the target DDL operation and a global timestamp of the target DDL operation in a log file for storing a second log record of the target data table.

[0016] Optionally, in a specific implementation, before traversing each database instance of the distributed database, the method further includes:

[0017] Based on a pre-established mapping relationship between a sub-table and a data table, a target sub-table belonging to the target data table is determined.

[0018] Optionally, in a specific implementation, traversing each database instance of the distributed database includes:

[0019] After receiving the specified operation instruction, the computing node of the distributed database traverses each database instance of the distributed database;

[0020] The specified operation instruction is an operation instruction for the target DDL operation on the target data table.

[0021] In a second aspect, an embodiment of the present invention provides a DDL operation synchronization device, the device comprising:

[0022] A log record acquisition module, used to traverse each database instance of the distributed database, and when traversing to each database instance, obtain a first log record for a target subtable in the database instance; wherein the target subtable is a subtable of a target data table in a centralized database; and the first log record is used to record a target DDL operation performed on the target subtable;

[0023] A target record detection module is used to detect whether there is a target record matching the first log record in a log file for storing a second log record of the target data table; if so, traverse the next database instance and trigger the log record acquisition module; if not, trigger the operation execution module; wherein the second log record is used to record the DDL operation performed on the target data table; the DDL operation recorded in the target record is the target DDL operation, and the occurrence time information of the recorded DDL operation matches the occurrence time information of the target DDL operation;

[0024] The operation execution module is used to add the target record in the log file, execute the target DDL operation on the target data table, and traverse the next database instance to trigger the log record acquisition module.

[0025] Optionally, in a specific implementation, the first log record includes operation information of the target DDL operation performed on the target subtable, and a global timestamp corresponding to the target DDL operation; the second log record includes operation information of the DDL operation performed on the target data table, and a global timestamp corresponding to the performed DDL operation; wherein the global timestamp corresponding to any DDL operation is: a timestamp set for the DDL operation by the computing node of the distributed database after receiving the operation instruction of the DDL operation; the target record detection module is specifically used to:

[0026] It is detected whether there is a target record including operation information of the target DDL operation and a global timestamp of the target DDL operation in a log file for storing a second log record of the target data table.

[0027] Optionally, in a specific implementation, the device further includes:

[0028] The target sub-table determination module is used to determine the target sub-table belonging to the target data table based on the pre-established mapping relationship between the sub-table and the data table before traversing each database instance of the distributed database.

[0029] Optionally, in a specific implementation, the log record acquisition module is specifically used to:

[0030] After receiving the designated operation instruction, the computing node of the distributed database traverses each database instance of the distributed database; wherein the designated operation instruction is an operation instruction for the target DDL operation on the target data table.

[0031] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0032] Memory, used to store computer programs;

[0033] The processor is used to implement the steps of any DDL operation synchronization method provided in the first aspect when executing the program stored in the memory.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any DDL operation synchronization method provided in the first aspect are implemented.

[0035] In a fifth aspect, an embodiment of the present invention provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of any one of the DDL operation synchronization methods provided in the first aspect.

[0036] Beneficial effects of the embodiments of the present invention:

[0037] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, when performing a target DDL operation on each target sub-table belonging to the target data table in the distributed database and synchronizing the target DDL operation to the target data table in the centralized database, each database instance of the distributed database can be traversed, and when traversing each database instance, the first log record for the target sub-table in the database instance is obtained. Furthermore, it can be detected whether there is a target record matching the above first log record in the log file used to store the second log record of the target data table.

[0038] Among them, if the above detection result is existence, it means that the target DDL operation for the target subtable in the database instance has been synchronized to the target data table in the centralized database, so there is no need to perform the target DDL operation on the target data table again, and then the next database instance can be traversed. If the above detection result is non-existent, it means that the target DDL operation for the target subtable in the database instance has not been synchronized to the target data table, so the above target record can be added to the above log file, and the target DDL operation can be performed on the target data table, and then the next database instance can be traversed.

[0039] In this way, during the synchronization of DDL operations, it is possible to traverse each database instance in the distributed database to detect whether a target subtable has synchronized the target DDL operation to the target data table in the centralized database. If so, no further synchronization is performed. Based on this, the problem of repeated DDL operations caused by the same DDL operations of multiple subtables belonging to the same data table in the distributed database being synchronized to the data table in the centralized database can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0041] Figure 1 A schematic diagram of a specific example of applying the DDL operation synchronization method provided by an embodiment of the present invention;

[0042] Figure 2 A flowchart of a DDL operation synchronization method provided by an embodiment of the present invention;

[0043] Figure 3 A flowchart of another DDL operation synchronization method provided by an embodiment of the present invention;

[0044] Figure 4 A flowchart of another DDL operation synchronization method provided by an embodiment of the present invention;

[0045] Figure 5 A flowchart of another DDL operation synchronization method provided by an embodiment of the present invention;

[0046] Figure 6 A structural diagram of a DDL operation synchronization device provided by an embodiment of the present invention;

[0047] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0049] After each sub-table stored in each database instance of the distributed database is migrated to the centralized database, each database instance can still store each sub-table. In addition, users can perform DDL (Data Definition Language) operations on these sub-tables, and synchronize the performed DDL operations to the corresponding data tables in the centralized database. In the related art, during the synchronization process of the DDL operation, the same DDL operations of multiple sub-tables belonging to the same data table in the distributed database will be synchronized to the data table in the centralized database respectively. In this way, the problem of repeated DDL operations will occur.

[0050] In order to make it easier to understand the technical problem to be solved by a DDL operation synchronization method provided in an embodiment of the present invention, the following describes a specific example to illustrate the problem of repeated DDL operations caused by the same DDL operations of multiple sub-tables belonging to the same data table in a distributed database being synchronized to the data table in a centralized database during the synchronization process of the DDL operations.

[0051] Assume that subtable X of data table A is stored in database instance 1 of the distributed database, subtable Y of data table A is stored in database instance 2 of the distributed database, and data table A is stored in the centralized database. When the same table creation operation is first performed on subtable X and subtable Y in the distributed database, and then the same table deletion operation is performed, the following situation may occur:

[0052] Step 1: The table creation operation of sub-table X is synchronized to data table A in the centralized database, and the table creation operation of data table A in the centralized database is completed;

[0053] Step 2: The deletion operation of sub-table X is synchronized to data table A in the centralized database, and then data table A in the centralized database completes the table deletion operation, deleting the table created in step 1;

[0054] Step 3: The table creation operation of sub-table Y is synchronized to the data table A in the centralized database. Then, the table creation operation of the data table A in the centralized database is completed, and the table created in step 1 is created again.

[0055] Step 4: The deletion operation of sub-table Y is synchronized to data table A in the centralized database, and then data table A in the centralized database completes the table deletion operation and deletes the table created in step 3.

[0056] Obviously, in the above process, the data table A in the centralized database will continuously repeat the table creation and deletion operations, so that the same table is continuously created and deleted. In this way, due to the timing problem of each DDL operation synchronized to the data table A in the centralized database, the data table A in the centralized database will have the problem of repeated DDL operations, so that the synchronization of the DDL operations of the distributed database and the centralized database cannot be completed normally.

[0057] In order to solve the above technical problem, an embodiment of the present invention provides a DDL operation synchronization method.

[0058] Among them, this method is applicable to various application scenarios that require synchronizing the DDL operations of each sub-table in a distributed database to the data table to which each sub-table belongs in a centralized database, for example, using a centralized database to back up data stored in a distributed database, etc., of course, it is not limited to this.

[0059] The method is applied to a data migration (Data Transformation Service, DTS) tool, wherein the data migration tool is a data transformation service that can realize the movement of data from one storage location to another storage location, and can also synchronize various operations on data stored in one storage location to corresponding data stored in another storage location. In a specific application, the data migration tool used to execute the method is a data migration tool pre-set in various electronic devices such as laptops and desktop computers, and of course, it can also be a data migration tool pre-set in a device cluster.

[0060] The DDL operation synchronization method provided by the embodiment of the present invention may include the following steps:

[0061] Traversing each database instance of the distributed database, when traversing to each database instance, obtaining a first log record for a target subtable in the database instance; wherein the target subtable is a subtable of a target data table in a centralized database; and the first log record is used to record a target DDL operation performed on the target subtable;

[0062] Detecting whether there is a target record matching the first log record in a log file for storing a second log record of the target data table; wherein the second log record is used to record a DDL operation performed on the target data table; the DDL operation recorded in the target record is the target DDL operation, and the occurrence time information of the recorded DDL operation matches the occurrence time information of the target DDL operation;

[0063] If it exists, traverse the next database instance;

[0064] If it does not exist, the target record is added to the log file, the target DDL operation is performed on the target data table, and the next database instance is traversed.

[0065] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, when performing a target DDL operation on each target sub-table belonging to the target data table in the distributed database and synchronizing the target DDL operation to the target data table in the centralized database, each database instance of the distributed database can be traversed, and when traversing each database instance, the first log record for the target sub-table in the database instance is obtained. Furthermore, it can be detected whether there is a target record matching the above first log record in the log file used to store the second log record of the target data table.

[0066] Among them, if the above detection result is existence, it means that the target DDL operation for the target subtable in the database instance has been synchronized to the target data table in the centralized database, so there is no need to perform the target DDL operation on the target data table again, and then the next database instance can be traversed. If the above detection result is non-existent, it means that the target DDL operation for the target subtable in the database instance has not been synchronized to the target data table, so the above target record can be added to the above log file, and the target DDL operation can be performed on the target data table, and then the next database instance can be traversed.

[0067] In this way, during the synchronization of DDL operations, it is possible to traverse each database instance in the distributed database to detect whether a target subtable has synchronized the target DDL operation to the target data table in the centralized database. If so, no further synchronization is performed. Based on this, the problem of repeated DDL operations caused by the same DDL operations of multiple subtables belonging to the same data table in the distributed database being synchronized to the data table in the centralized database can be solved.

[0068] Before specifically describing a DDL operation synchronization method provided by an embodiment of the present invention, a brief description is first given of the storage method of data in a centralized database and a distributed database.

[0069] The data stored in the centralized database is stored in the same data storage space. Therefore, the centralized database presents a unified data logic table to the outside world. That is to say, when the user responds to data access based on the centralized database, what the user sees is the unified data logic table.

[0070] Accordingly, the distributed database has multiple independent database instances, and each piece of data is stored in a different database instance. In addition, the data stored in each database instance is stored according to the designated data sub-tables, that is, each piece of data stored in the same database instance is stored in different sub-tables in the database instance. Thus, each database instance separately presents to the outside the data sub-tables to which the stored data belongs, and the data logical table to which each sub-table belongs, wherein the data sub-tables belonging to the same data logical table can be located in different database instances.

[0071] For example, Figure 1 As shown, TableA is a data logic table, subtable X and subtable Y are data subtables belonging to TableA in database instance 1 (DB1), and subtable M and subtable N are data subtables belonging to TableA in database instance N (DBN). DB is the abbreviation of Database.

[0072] Optionally, the distributed database may generally be a distributed data database system including a plurality of data storage nodes, wherein each data storage node is a database instance.

[0073] Typically, a node can be set up in a distributed database as a computing node (proxy) of the distributed database, wherein the computing node can receive operation instructions for DDL operations on a certain data logic table sent by an application client, so that the computing node can perform the above-mentioned DDL operations on each sub-table belonging to the data logic table stored in each database instance of the distributed database.

[0074] The computing node may be a separate node from each database instance, or may be a database instance in each data storage node. The database instance may be used to execute DDL operations and to store data.

[0075] A DDL operation synchronization method provided by an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0076] Figure 2 A flowchart of a DDL operation synchronization method provided by an embodiment of the present invention is shown as follows: Figure 2As shown, the method may include the following steps S201-S204.

[0077] S201: traversing each database instance of the distributed database, and when traversing to each database instance, obtaining a first log record for a target subtable in the database instance;

[0078] The target subtable is a subtable of the target data table in the centralized database; the first log record is used to record the target DDL operation performed on the target subtable;

[0079] When synchronizing the DDL operations of each sub-table in the distributed database to the data table to which each sub-table belongs in the centralized database, the data table to be subjected to the DDL operation in the centralized database can be used as the target data table, and then the target sub-table belonging to the target data table in each database instance of the distributed database can be determined. That is, the target sub-table in each database instance of the distributed database is: a sub-table of the target data table in the centralized database.

[0080] It should be emphasized that the target data table does not specifically refer to a data table in the centralized database, but can be any data table in the centralized database, or any multiple data tables in the centralized database. Furthermore, when the target data table is any multiple data tables in the centralized database, the target sub-tables belonging to each target data table in each database instance of the distributed database can be determined respectively.

[0081] In this way, each database instance of the distributed database can be traversed, and when each database instance is traversed, the first log record for the target sub-table in the database instance can be obtained.

[0082] Among them, the above-mentioned first log record records the target DDL operation performed on the target sub-table. Therefore, when completing the DDL operation synchronization of the distributed database and the centralized database, the target DDL operation performed on the target sub-table needs to be synchronized to the target data table in the centralized database.

[0083] Optionally, the target DDL operation may be: deleting a column in a data table, adding a column in a data table, modifying a column in a data table, deleting a table, and other DDL operations, but the present invention is not limited thereto.

[0084] S202: Detect whether there is a target record matching the first log record in the log file for storing the second log record of the target data table; if so, traverse the next database instance; if not, execute S203;

[0085] The second log record is used to record the DDL operation performed on the target data table; the DDL operation recorded in the target record is the target DDL operation, and the occurrence time information of the recorded DDL operation matches the occurrence time information of the target DDL operation;

[0086] S203: Add a target record to the log file, execute a target DDL operation on the target data table, and traverse the next database instance.

[0087] After traversing each database instance and obtaining the first log record for the target subtable in the database instance, it is possible to detect whether there is a target record matching the first log record in the log file for storing the second log record of the target data table.

[0088] The second log record is used to record the DDL operation performed on the target data table in the centralized database, that is, the log file records the DDL operation performed on the target data table in the centralized database.

[0089] In this way, since the DDL operation recorded in the target record is: the above-mentioned target DDL operation, and the occurrence time information of the DDL operation recorded in the target record matches the occurrence time information of the above-mentioned target DDL operation, the above-mentioned target record is used to record the target DDL operation performed on the target data table in the centralized database at a time that matches the operation time of performing the target DDL operation on the target sub-table in the traversed database instance.

[0090] Based on this, when the above target record is recorded in the above log file, it can be explained that:

[0091] For the traversed database instance, the target DDL operation performed on the target subtable in the database instance has been synchronized to the target data table in the centralized database. In this way, there is no need to synchronize the target DDL operation to the target data table in the centralized database again, so that the first log record obtained for the target subtable in the database instance can be ignored to avoid repeating the target DDL operation to the target data table in the centralized database, and the next database instance is traversed to obtain the first log record for the target subtable in the next database instance, until the last database instance is traversed, and a DDL operation synchronization is completed.

[0092] Correspondingly, when the above target record is not recorded in the above log file, it can be explained that:

[0093] For the traversed database instance, the target DDL operation performed on the target subtable in the database instance has not yet been synchronized to the target data table in the centralized database. In this way, in order to achieve DDL operation synchronization between the distributed database and the centralized database, it is necessary to synchronize the target DDL operation to the target data table in the centralized database, that is, to perform the above target DDL operation on the target data table in the centralized database, and add the target record to the above log file, so that when the target record is searched for again in the log file later, it can be found, thereby avoiding the subsequent duplication of the target DDL operation to the target data table in the centralized database.

[0094] After adding the target record to the log file and executing the target DDL operation on the target data table, the next database instance can be traversed to obtain the first log record for the target subtable in the next database instance until the last database instance is traversed. At this point, a DDL operation synchronization is completed.

[0095] Optionally, the log file may be stored in a centralized database or in other storage devices that can communicate with the data migration tool. This is not specifically limited in the embodiments of the present invention.

[0096] Further, optionally, since the above-mentioned log file may change after traversing each database instance, after obtaining the above-mentioned first log record and before detecting whether there is a target record matching the first log record in the log file used to store the second log record of the target data table, the currently stored log file can be determined first, thereby detecting whether there is a target record matching the first log record in the currently stored log file.

[0097] In this way, in the process of traversing each database instance in the distributed database, each target DDL operation of the target data table that has been synchronized to the centralized database can be directly ignored. Therefore, it is equivalent to automatically recovering the first log record of the target sub-table of each database instance that is expired and invalid according to the synchronization progress of the target DDL operation of the target sub-table in each database instance in the process of traversing each database instance in the distributed database. Then, each target DDL operation for the target data table in the centralized database can be automatically deduplicated, and the above-mentioned log file can be maintained to ensure the uniqueness of the DDL operations performed on the target data table recorded in the above-mentioned log file.

[0098] Based on this, by applying the solution provided by the embodiment of the present invention, during the synchronization process of the DDL operation, it is possible to traverse each database instance in the distributed database to detect whether a certain target subtable has synchronized the target DDL operation to the target data table in the centralized database. If so, no further synchronization is performed. Based on this, the problem of repeated DDL operations caused by the same DDL operations of multiple subtables belonging to the same data table in the distributed database being synchronized to the data table in the centralized database can be solved.

[0099] Optionally, in a specific implementation, the first log record includes operation information of a target DDL operation performed on the target sub-table, and a global timestamp corresponding to the target DDL operation;

[0100] The second log record includes operation information of the DDL operation performed on the target data table and a global timestamp corresponding to the performed DDL operation;

[0101] The global timestamp corresponding to any DDL operation is: the timestamp set by the computing node of the distributed database for the DDL operation after receiving the operation instruction of the DDL operation;

[0102] In this specific implementation, when traversing each database instance, the first log record obtained for the target subtable in the database instance may include: operation information of the target DDL operation performed on the target subtable in the database instance, and the global timestamp corresponding to the target DDL operation.

[0103] Similarly, the second log record for recording the DDL operation performed on the target data table may include: operation information of the DDL operation performed on the target data table, and a global timestamp corresponding to the performed DDL operation.

[0104] Among them, whether in the above-mentioned first log record or the above-mentioned second log record, the global timestamp corresponding to any DDL operation contained therein can be: the timestamp set for the DDL operation by the computing node of the distributed database after receiving the operation instruction of the DDL operation.

[0105] Generally, a distributed database may have a node set as a management node of the distributed database, for example, Figure 1 As shown, the global management node (GMS) is a management node. The management node can be a separate node from each database instance, or a database instance in each data storage node. The database instance can be used to manage the distributed database and store data.

[0106] In this way, the computing node can receive the operation instruction of the DDL operation for a certain data logic table sent by the application client, and then the computing node can obtain the timestamp of the DDL operation specified by the operation instruction from the management node. Thus, the computing node can set the obtained timestamp as the global timestamp of the DDL operation specified by the operation instruction.

[0107] Based on this, the DDL operation for each sub-table in each database instance of the distributed database can carry a globally unique timestamp identifier that increases monotonically with the execution time. Therefore, the log record generated for each sub-table can carry the timestamp identifier of the DDL operation.

[0108] Accordingly, in this specific implementation, if Figure 3 As shown, the above step S202, detecting whether there is a target record matching the first log record in the log file for storing the second log record of the target data table, may include the following step S2021:

[0109] S2021: Detect whether there is a target record containing operation information of the target DDL operation and a global timestamp of the target DDL operation in the log file for storing the second log record of the target data table.

[0110] In this specific implementation, since the first log record contains the operation information of the target DDL operation performed on the target sub-table and the global timestamp corresponding to the target DDL operation, in the log file for storing the second log record of the target data table, the target record to be detected is the record containing the operation information of the target DDL operation and the global timestamp of the target DDL operation.

[0111] That is to say, after traversing each database instance and obtaining the first log record containing the operation information of the target DDL operation performed on the target sub-table and the global timestamp corresponding to the target DDL operation, it is possible to detect whether there is a target record containing the operation information of the target DDL operation and the global timestamp of the target DDL operation in the log file used to store the second log record of the target data table.

[0112] Optionally, in a specific implementation, such as Figure 4 As shown, a DDL operation synchronization method provided by an embodiment of the present invention may also include the following step S200:

[0113] S200: Determine a target sub-table belonging to a target data table based on a pre-established mapping relationship between a sub-table and a data table.

[0114] In this specific implementation, when synchronizing the DDL operations of each sub-table in the distributed database to the data table to which each sub-table belongs in the centralized database, the target data table to be subjected to the DDL operation in the centralized database can be first determined, thereby traversing each database instance in the distributed database to obtain the first log record for the target sub-table belonging to the target data table in the database instance.

[0115] Since each database instance in the distributed database may store subtables belonging to multiple different data tables, before traversing each database instance in the distributed database, the target subtable belonging to the target data table may be determined based on the pre-established mapping relationship between the subtable and the data table. Thus, when traversing each database instance, the target subtable belonging to the target data table may be determined among the subtables stored in the database instance, and then the first log record for the target subtable in the database instance may be obtained.

[0116] Based on this, in this specific implementation, the data migration tool can automatically complete the mapping relationship between different sub-tables belonging to the same data table and the data table to which they belong in different database instances in the distributed database.

[0117] Optionally, in a specific implementation, in order to ensure the synchronization timeliness of DDL operations of the distributed database and the centralized database, the above-mentioned specific implementations can be performed in real time.

[0118] Optionally, in a specific implementation, considering that the DDL operations on the sub-tables in each database instance of the distributed database may not be performed in real time, in order to save resource consumption during the execution of each of the above specific implementations, each of the above specific implementations may also be performed periodically.

[0119] Optionally, in a specific implementation, such as Figure 5 As shown, the above step S201 may include the following step S2011:

[0120] S2011: After receiving the specified operation instruction, the computing node of the distributed database traverses each database instance of the distributed database;

[0121] The specified operation instruction is an operation instruction for a target DDL operation on a target data table.

[0122] In this specific implementation, the data migration tool can detect the computing nodes of the distributed database to determine whether the computing nodes of the distributed database have received the specified operation instructions, that is, the data migration tool can monitor whether the computing nodes of the distributed database have received the operation instructions for the target DDL operation of the target data table.

[0123] In this way, in order to ensure the synchronization timeliness of DDL operations between the distributed database and the centralized database, and to avoid the resource consumption caused by executing the above-mentioned specific implementation methods when there are no DDL operations on the sub-tables in each database instance of the distributed database, it is possible to traverse each database instance of the distributed database after the computing node of the distributed database receives the above-mentioned specified operation instructions.

[0124] That is to say, after receiving the above-mentioned designated operation instruction, the computing node of the distributed database can execute the above-mentioned specific implementation methods to realize the synchronization of the DDL operations of the distributed database and the centralized database. In this way, while ensuring the synchronization timeliness of the DDL operations of the distributed database and the centralized database, it is possible to avoid the resource consumption caused by executing the above-mentioned specific implementation methods when there is no DDL operation for the sub-tables in each database instance of the distributed database.

[0125] To facilitate understanding of the DDL operation synchronization method provided by the embodiment of the present invention, Figure 1 The specific example shown illustrates the DDL operation synchronization method provided by the embodiment of the present invention.

[0126] like Figure 1 As shown, a distributed database is provided with multiple data block instances DB1-DBN, computing nodes (proxy) and a global management unit (GMS). Each database instance may store each subtable of Table (data table) A, for example, DB1 stores subtable X and subtable Y of Table A, and DBN stores subtable M and subtable N of Table A. The computing nodes communicate with the application APP (Application, client) and the global management unit respectively.

[0127] Furthermore, the target database DB is a centralized database; the only content that interacts with the data migration tool (DTS) is the log file for storing the second log record of the target data table.

[0128] In this way, Figure 1 In step 1: DDL on Table A, the application APP sends an operation instruction for the DDL operation of Table A to the computing node in the distributed database. Then, the computing node in the distributed database can receive the above operation instruction and determine the DDL operation indicated by the above operation instruction.

[0129] like Figure 1In step 2: Get DDL timestamp As TS shown in the figure, the computing node in the distributed database can obtain the global timestamp TS corresponding to the DDL operation indicated by the above operation instruction from the global management unit, and set the global timestamp TS for the DDL operation.

[0130] like Figure 1 As shown in step 3.1: DDL on Table A sub-table X DDL on Table A sub-table Y, and step 3.2: DDL on Table A sub-table M DDL on Table A sub-table N, the computing nodes in the distributed database can perform the above DDL operations on each sub-table belonging to Table A in each database instance of the distributed database, and record the first log record for each sub-table belonging to Table A in the database instance in the log of each database instance, and the first log record records the operation information of the above DDL operation and the global timestamp TS corresponding to the above DDL operation. Wherein, Figure 1 Steps 3.1 and 3.2 shown in the figure are merely examples, not limitations, of a computing node in a distributed database performing the above DDL operations on each sub-table belonging to Table A in each database instance of the distributed database.

[0131] like Figure 1 In step 5: DDL on Table A sub-table X DDL on Table A sub-table Y with TS, the data migration tool can traverse DB1-DBN. When traversing to DB1, the data migration tool obtains the first log record for sub-table X and sub-table Y belonging to Table A in DB1, wherein the first log record includes: operation information of the above DDL operation on sub-table X and sub-table Y and the global timestamp TS corresponding to the above DDL operation.

[0132] The data migration tool detects whether there is a target record including the occurrence time information TS and the operation information of the above DDL operation in the log file for storing the second log record of Table A.

[0133] like Figure 1 As shown in step 6: record the DDL operation of Table A at time TS, and step 7: DDL on TableA, if the above detection result is not present, the data migration tool performs the above DDL operation on Table A in the centralized database, and adds a target record containing the occurrence time information TS and the operation information of the above DDL operation to the above log file.

[0134] The data migration tool continues to traverse DB2. When traversing to DB2, the data migration tool obtains the first log record for the subtable belonging to Table A in DB2, wherein the first log record includes: operation information of the above DDL operation for the subtable belonging to Table and the global timestamp TS corresponding to the above DDL operation.

[0135] The data migration tool detects whether there is a target record including the occurrence time information TS and the operation information of the above DDL operation in the log file for storing the second log record of Table A.

[0136] If the detection result is yes, the data migration tool ignores the DDL operation to avoid repeating the DDL operation on Table A in the centralized database. Then, the data migration tool continues to traverse DB3.

[0137] And so on, until the DBN is traversed, such as Figure 1 As shown in step 8: DDL on Table A sub-table M DDL on Table A sub-table N with TS, and step 9: if it is found that the DDL operation of Table A at time TS has been carried out, then the DDL at time TS is ignored, and the data migration tool obtains the first log record for the sub-table belonging to Table A in DBN, wherein the first log record contains: the operation information of the above DDL operation for the sub-table belonging to Table and the global timestamp TS corresponding to the above DDL operation. The data migration tool detects whether there is a target record containing the occurrence time information TS and the operation information of the above DDL operation in the log file used to store the second log record of Table A.

[0138] If the detection result is yes, the data migration tool ignores the DDL operation to avoid repeating the DDL operation on Table A in the centralized database. At this point, the DDL operation synchronization is completed.

[0139] Corresponding to the DDL operation synchronization method provided by the above-mentioned embodiment of the present invention, the embodiment of the present invention also provides a DDL operation synchronization device.

[0140] Figure 6 A DDL operation synchronization device is provided in an embodiment of the present invention, such as Figure 6 As shown, the device may include the following modules:

[0141] The log record acquisition module 610 is used to traverse each database instance of the distributed database, and when traversing to each database instance, obtain a first log record for a target subtable in the database instance; wherein the target subtable is a subtable of a target data table in the centralized database; and the first log record is used to record a target DDL operation performed on the target subtable;

[0142] The target record detection module 620 is used to detect whether there is a target record matching the first log record in the log file for storing the second log record of the target data table; if so, traverse the next database instance and trigger the log record acquisition module; if not, trigger the operation execution module; wherein the second log record is used to record the DDL operation performed on the target data table; the DDL operation recorded in the target record is the target DDL operation, and the occurrence time information of the recorded DDL operation matches the occurrence time information of the target DDL operation;

[0143] The operation execution module 630 is used to add the target record to the log file, execute the target DDL operation on the target data table, and traverse the next database instance to trigger the log record acquisition module.

[0144] Based on this, by applying the solution provided by the embodiment of the present invention, during the synchronization process of the DDL operation, it is possible to traverse each database instance in the distributed database to detect whether a certain target subtable has synchronized the target DDL operation to the target data table in the centralized database. If so, no further synchronization is performed. Based on this, the problem of repeated DDL operations caused by the same DDL operations of multiple subtables belonging to the same data table in the distributed database being synchronized to the data table in the centralized database can be solved.

[0145] Optionally, in a specific implementation, the first log record includes operation information of the target DDL operation performed on the target subtable, and a global timestamp corresponding to the target DDL operation; the second log record includes operation information of the DDL operation performed on the target data table, and a global timestamp corresponding to the performed DDL operation; wherein the global timestamp corresponding to any DDL operation is: a timestamp set for the DDL operation by the computing node of the distributed database after receiving the operation instruction of the DDL operation; the target record detection module 620 is specifically used to:

[0146] It is detected whether there is a target record including operation information of the target DDL operation and a global timestamp of the target DDL operation in a log file for storing a second log record of the target data table.

[0147] Optionally, in a specific implementation, the device further includes:

[0148] The target sub-table determination module is used to determine the target sub-table belonging to the target data table based on the pre-established mapping relationship between the sub-table and the data table before traversing each database instance of the distributed database.

[0149] Optionally, in a specific implementation, the log record acquisition module 610 is specifically used to:

[0150] After receiving the designated operation instruction, the computing node of the distributed database traverses each database instance of the distributed database; wherein the designated operation instruction is an operation instruction for the target DDL operation on the target data table.

[0151] Corresponding to the DDL operation synchronization method provided by the above embodiment of the present invention, the embodiment of the present invention also provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0152] Memory 703, used for storing computer programs;

[0153] The processor 701 is configured to implement the steps of any DDL operation synchronization method provided by the above-mentioned embodiments of the present invention when executing the program stored in the memory 703 .

[0154] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0155] The communication interface is used for communication between the above electronic device and other devices.

[0156] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0157] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0158] In another embodiment provided by the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any DDL operation synchronization method provided by the above-mentioned embodiments of the present invention are implemented.

[0159] In another embodiment provided by the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute the steps of any DDL operation synchronization method provided by the above-mentioned embodiments of the present invention.

[0160] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0161] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0162] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0163] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A DDL operation synchronization method, characterized in that: The method comprises: Traversing each database instance of the distributed database, when traversing to each database instance, obtaining a first log record for a target subtable in the database instance; wherein the target subtable is a subtable of a target data table in a centralized database; and the first log record is used to record a target DDL operation performed on the target subtable; Detecting whether there is a target record matching the first log record in a log file for storing a second log record of the target data table; wherein the second log record is used to record a DDL operation performed on the target data table; the DDL operation recorded in the target record is the target DDL operation, and the occurrence time information of the recorded DDL operation matches the occurrence time information of the target DDL operation; If it exists, traverse the next database instance; If it does not exist, the target record is added to the log file, the target DDL operation is performed on the target data table, and the next database instance is traversed.

2. The method according to claim 1, characterized in that The first log record includes operation information of a target DDL operation performed on the target sub-table and a global timestamp corresponding to the target DDL operation; The second log record includes operation information of the DDL operation performed on the target data table and a global timestamp corresponding to the performed DDL operation; The global timestamp corresponding to any DDL operation is: the timestamp set for the DDL operation by the computing node of the distributed database after receiving the operation instruction of the DDL operation; The detecting whether there is a target record matching the first log record in a log file for storing the second log record of the target data table includes: It is detected whether there is a target record including operation information of the target DDL operation and a global timestamp of the target DDL operation in a log file for storing a second log record of the target data table.

3. The method according to claim 1, characterized in that Before traversing each database instance of the distributed database, the method further includes: Based on a pre-established mapping relationship between a sub-table and a data table, a target sub-table belonging to the target data table is determined.

4. The method according to claim 1, characterized in that: The traversing each database instance of the distributed database includes: After receiving the specified operation instruction, the computing node of the distributed database traverses each database instance of the distributed database; The specified operation instruction is an operation instruction for the target DDL operation on the target data table.

5. A DDL operation synchronization device, characterized in that: The device comprises: A log record acquisition module, used to traverse each database instance of the distributed database, and when traversing to each database instance, obtain a first log record for a target subtable in the database instance; wherein the target subtable is a subtable of a target data table in a centralized database; and the first log record is used to record a target DDL operation performed on the target subtable; A target record detection module is used to detect whether there is a target record matching the first log record in a log file for storing a second log record of the target data table; if so, traverse the next database instance and trigger the log record acquisition module; if not, trigger the operation execution module; wherein the second log record is used to record the DDL operation performed on the target data table; the DDL operation recorded in the target record is the target DDL operation, and the occurrence time information of the recorded DDL operation matches the occurrence time information of the target DDL operation; The operation execution module is used to add the target record in the log file, execute the target DDL operation on the target data table, and traverse the next database instance to trigger the log record acquisition module.

6. The device according to claim 5, characterized in that The first log record includes operation information of a target DDL operation performed on the target subtable and a global timestamp corresponding to the target DDL operation; the second log record includes operation information of a DDL operation performed on the target data table and a global timestamp corresponding to the performed DDL operation; wherein the global timestamp corresponding to any DDL operation is: a timestamp set for the DDL operation by the computing node of the distributed database after receiving the operation instruction of the DDL operation; the target record detection module is specifically used for: It is detected whether there is a target record including operation information of the target DDL operation and a global timestamp of the target DDL operation in a log file for storing a second log record of the target data table.

7. The device according to claim 5, characterized in that The device also includes: The target sub-table determination module is used to determine the target sub-table belonging to the target data table based on the pre-established mapping relationship between the sub-table and the data table before traversing each database instance of the distributed database.

8. The device according to claim 5, characterized in that The log record acquisition module is specifically used for: After receiving the designated operation instruction, the computing node of the distributed database traverses each database instance of the distributed database; wherein the designated operation instruction is an operation instruction for the target DDL operation on the target data table.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Images