Database synchronization processing method and apparatus, electronic device, and storage medium

Abstract

The application provides a database synchronization processing method and device, electronic equipment and storage medium, and relates to database technology and big data technology. The method comprises the following steps: obtaining metadata information of a source table in a source database and metadata information of a target table in a target database; checking consistency between a table structure of the source table and a table structure of the target table according to the metadata information of the source table and the metadata information of the target table; if the table structure of the source table is consistent with the table structure of the target table, querying a consumption offset record table corresponding to an operation record file of the source database to determine whether the table structure of the source table has changed; and if the table structure of the source table has not changed, synchronizing data of the source table to the target table. The application solves the problem that data synchronization cannot guarantee effectiveness and consumes high cost in the prior art.

Description

Technical Field

[0001] This application relates to database technology and big data technology, and in particular to a database synchronization processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, software development often faces the challenge of synchronizing data between different or identical databases. For example, scripts can be used to periodically export data from a source database and then import it into a target database for synchronization. However, this approach fails to meet users' demands for timely data synchronization.

[0003] Furthermore, during development, fields may be added or modified in the business-based tables of the source database. These changes need to be synchronized in the target database, which requires manual intervention and is too costly. In addition, systems developed for enterprise users have high security requirements, necessitating regular upgrades and remediation of relevant synchronization components to fix vulnerabilities, making maintenance too costly.

[0004] Therefore, existing data synchronization methods between databases generally suffer from problems such as ineffectiveness and excessive cost. Summary of the Invention

[0005] This application provides a database synchronization processing method, apparatus, electronic device, and storage medium to solve the problems of ineffective data synchronization and excessive cost in the prior art, thereby achieving real-time data synchronization between databases and reducing costs.

[0006] On the one hand, this application provides a database synchronization processing method, including:

[0007] Obtain metadata information of the source table in the source database and metadata information of the target table in the target database;

[0008] Based on the metadata information of the source table and the metadata information of the target table, verify the consistency between the table structure of the source table and the table structure of the target table;

[0009] If the table structure of the source table is consistent with the table structure of the target table, then query the consumption offset record table corresponding to the operation record file of the source database to determine whether the table structure of the source table has changed.

[0010] If the table structure of the source table has not changed, the data in the source table will be synchronized to the target table.

[0011] Further, querying the consumption offset record table corresponding to the operation record file in the source database to determine whether the table structure of the source table has changed includes:

[0012] Query the consumption offset record table corresponding to the operation record file in the source database to obtain the consumption position saved after the last successful consumption of the operation record file. The operation record file is used to record the data operation type and table structure change information for updating the source table.

[0013] Starting from the consumption location, the operation log file is parsed to determine whether the table structure of the source table has changed.

[0014] Furthermore, the method also includes:

[0015] If the table structure of the source table changes and the automatic table structure change function of the target table is enabled, then the table structure of the target table is changed based on the table structure of the source table.

[0016] Update the consumption offset record table corresponding to the operation log file;

[0017] Starting from the next position after the last successful consumption of the operation log file, the operation log file is parsed to determine whether the table structure of the source table has changed.

[0018] Furthermore, the method also includes:

[0019] If the table structure of the source table changes and the automatic table structure change function of the target table is turned off, then the consistency between the table structure of the source table and the table structure of the target table is checked again.

[0020] If the table structure of the source table is inconsistent with the table structure of the target table, wait for a period of time to continue the verification until the number of verifications exceeds the maximum number of retries, then stop the data synchronization processing program and output an error message.

[0021] If the table structure of the source table is consistent with the table structure of the target table, then the operation log file is parsed starting from the next position after the consumption position saved after the last successful consumption of the operation log file, in order to determine whether the table structure of the source table has changed.

[0022] Furthermore, the method also includes:

[0023] If the table structure of the source table is inconsistent with the table structure of the target table, the data synchronization process will stop and an error message will be output.

[0024] Furthermore, after synchronizing the data from the source table to the target table, the method further includes:

[0025] Starting from the next position after the last successful consumption of the operation log file, the operation log file is parsed to determine whether the table structure of the source table has changed.

[0026] Furthermore, the metadata information of both the source table and the target table includes: table field names, field types, primary keys, and field order.

[0027] On the other hand, this application provides a database synchronization processing apparatus, the apparatus comprising:

[0028] The acquisition module is used to acquire metadata information of the source table in the source database and metadata information of the target table in the target database.

[0029] The verification module is used to verify the consistency between the table structure of the source table and the table structure of the target table based on the metadata information of the source table and the metadata information of the target table.

[0030] The determination module is used to query the consumption offset record table corresponding to the operation record file of the source database if the table structure of the source table is consistent with the table structure of the target table, so as to determine whether the table structure of the source table has changed.

[0031] The synchronization module is used to synchronize the data of the source table to the target table if the table structure of the source table has not changed.

[0032] On the other hand, this application provides an electronic device, including: a processor and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement any of the methods described above.

[0033] On the other hand, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods described above.

[0034] On the other hand, this application provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.

[0035] The database synchronization method provided in this application obtains the metadata information of the source table in the source database and the metadata information of the target table in the target database; based on the metadata information of the source table and the target table, it verifies the consistency between the table structure of the source table and the table structure of the target table; if the table structure of the source table is consistent with the table structure of the target table, it queries the consumption offset record table corresponding to the operation record file of the source database to determine whether the table structure of the source table has changed; if the table structure of the source table has not changed, it synchronizes the data of the source table to the target table.

[0036] In this embodiment, an operation log file from the source database is used to record operation information of the source table, and a consumption offset record table is used to record the consumption address of the last consumption operation log file. Based on the metadata information of the source and target tables, the table structure of the source table is verified to be consistent with that of the target table. If the table structures are consistent, the consumption offset record table corresponding to the operation log file of the source database is queried to determine whether the table structure of the source table has changed. If the table structure of the source table has not changed, the data of the source table is directly synchronized to the target table. This ensures the timeliness of data synchronization between databases, requires no manual operation, has high security, and reduces the cost of fixing vulnerabilities. Therefore, this embodiment can solve the problems of ineffective data synchronization and excessive cost in the prior art. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1 A schematic flowchart illustrating a database synchronization method provided in an embodiment of this application;

[0039] Figure 2 A flowchart illustrating an optional database synchronization method provided in this application embodiment;

[0040] Figure 3 A flowchart illustrating an optional database synchronization method provided in an embodiment of this application;

[0041] Figure 4 A flowchart illustrating an optional database synchronization method provided in this application embodiment;

[0042] Figure 5 A flowchart illustrating an optional database synchronization method provided in an embodiment of this application;

[0043] Figure 6A structural block diagram of a database synchronization processing device provided in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0045] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] First, let me explain the terms used in this application:

[0048] MySQL is an open-source relational database.

[0049] Binlog is a binary file used to record SQL statements used by users to update the database. For example, SQL statements that modify tables and change content will be recorded in the binlog, but queries on database tables and other content will not be recorded.

[0050] In software development, the problem of synchronizing data between different or identical databases is frequently encountered. Synchronization methods can be categorized into batch offline synchronization and real-time synchronization. Consequently, various related data synchronization tools have emerged.

[0051] Currently, there are several relatively mature real-time data synchronization tools, including the open-source tool Canal, which can effectively achieve real-time data synchronization. However, it has two main drawbacks: 1. For scenarios with simple business requirements, its architecture is complex and resource consumption is high; 2. Due to its open-source nature, the official team does not actively upgrade the third-party open-source components that its code depends on, resulting in numerous security vulnerabilities and high maintenance costs.

[0052] In addition, other real-time synchronization tools focus on synchronizing data to a message broker and then consuming the data from the broker themselves. MySQL's built-in master-slave replication function requires the target database to be used as a slave database; however, the slave database needs to maintain the same data architecture as the master database and cannot provide services to the business.

[0053] The database synchronization processing method provided in this application aims to solve the above-mentioned technical problems of the prior art. The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0054] Figure 1 This is a flowchart illustrating a database synchronization method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0055] S101, obtain the metadata information of the source table in the source database and the metadata information of the target table in the target database.

[0056] S102, based on the metadata information of the source table and the metadata information of the target table, verify the consistency between the table structure of the source table and the table structure of the target table.

[0057] S103, if the table structure of the source table is consistent with the table structure of the target table, then query the consumption offset record table corresponding to the operation record file of the source database to determine whether the table structure of the source table has changed.

[0058] S104. If the table structure of the source table has not changed, then synchronize the data of the source table to the target table.

[0059] Optionally, the source database mentioned above is the upstream database, for example, the database that initially generated the data, and the target database mentioned above is the downstream database, which is the database to which the data initially generated in the source database needs to be synchronized.

[0060] Optionally, in one example, the metadata information of both the source table and the target table includes: table field names, field types, primary keys, and field order.

[0061] The database synchronization method provided in this application can be, but is not limited to, synchronizing data from a source database, such as a MySQL database, to other target databases in real time. This solution utilizes the open-source tool mysql-binlog-connector-java to monitor changes in the binlog file of the source database and obtain the data to be synchronized from the source table. After obtaining the data, it is manually inserted into the target table of another database. This can be understood as relying on the aforementioned functionality of the open-source tool mysql-binlog-connector-java to implement a portion of the solution in this application, rather than the overall business process and inventive concept.

[0062] Therefore, this application embodiment monitors the binlog file of the source database using open-source tools. When a business or task triggers an add, delete, or modify operation on the source table, a corresponding log record will be generated in the binlog of the source database. When an add, delete, or modify operation is detected in the source table, the SQL statement information of the add, delete, or modify operation can be restored based on the log record. Therefore, the real-time nature of data synchronization can be guaranteed.

[0063] It should be noted that the open-source tool mysql-binlog-connector-java is a monitoring component. This component can connect to the source database's (upstream database) operation log file (binlog) based on the connection information, and then parse the information in the binlog into data that meets certain rules. Using this component avoids redundant development of this function, allowing focus on the overall business functionality of the database synchronization method, thus facilitating the implementation of business functions. This application's embodiment relies on few open-source components, has a lightweight architecture, is secure, and is easy to upgrade and modify; the deployment package is small, and resource consumption is low.

[0064] In one alternative embodiment, the mapping relationship between the source table and the target table to be synchronized can be pre-configured in the configuration file. The source table and the target table need to be in different databases, for example, they can be the same or different database instances, but the database names are different.

[0065] Furthermore, the embodiments of this application are flexible in deployment. For example, by modifying the configuration file (XML format configuration file), the database connection information and the table names of the synchronized database tables can be configured to add or modify the required synchronized database tables, which facilitates the implementation of the solution.

[0066] After the data synchronization process starts, it first obtains the metadata information of the source table to be synchronized in the source database, such as table field names, field types, primary keys, field order, etc. Based on the metadata information of the source table, it verifies the consistency between the table structure of the source table and the table structure of the target table. The source table and the target table have the same table name to facilitate querying and comparing the consistency of the table structure.

[0067] For example, obtain the table structure (mainly table field names, field types, primary keys, field order, etc.) of the same table name in the source database and the target database respectively, and then verify whether the table structure information of the source table and the target table are consistent.

[0068] If inconsistencies occur, writing data from the source table in the source database into the target table in the target database will cause problems. Therefore, if inconsistencies are found, the data synchronization process should be stopped and an error message should be output.

[0069] It is understood that, in the embodiments of this application, since the data synchronization process may be stopped manually or accidentally, the data synchronization process must start from the last successful consumption position after restarting to ensure that the data is not lost and is not processed repeatedly.

[0070] Figure 2 A flowchart illustrating an optional database synchronization method provided in this application embodiment is shown below. Figure 2 As shown, if the source table and the target table have the same table structure, the consumption offset record table (also called the "consumption binlog offset record table") corresponding to the operation log file binlog of the source database is queried. This can determine the consumption position saved after the last successful consumption of the above operation log file. Starting from the above consumption position, the above operation log file is parsed, that is, the records in the operation log file are parsed one by one.

[0071] Since the operation log file (binlog) is generated based on the source table that needs to be synchronized, the content of the binlog is used to determine whether the table structure of the source table has changed, in order to perform synchronization processing. As before... Figure 2 As shown, if the source table does not undergo any changes in its table structure, for example, if the data operation type of the source table is a data addition, deletion, or modification operation, then the same addition, modification, or deletion operation will be performed in the corresponding target table in the target database. This achieves real-time synchronization of the data generated by the upstream source database to the downstream target database.

[0072] In this embodiment, an operation log file from the source database is used to record operation information of the source table, and a consumption offset record table is used to record the consumption address of the last consumption operation log file. Based on the metadata information of the source and target tables, the table structure of the source table is verified to be consistent with that of the target table. If the table structures are consistent, the consumption offset record table corresponding to the operation log file of the source database is queried to determine whether the table structure of the source table has changed. If the table structure of the source table has not changed, the data of the source table is directly synchronized to the target table. This ensures the timeliness of data synchronization between databases, requires no manual operation, has high security, and reduces the cost of fixing vulnerabilities. Therefore, this embodiment can solve the problems of ineffective data synchronization and excessive cost in the prior art.

[0073] In one example, Figure 3 A flowchart illustrating an optional database synchronization method provided in this application embodiment is shown below. Figure 3 As shown, the above query retrieves the consumption offset record table corresponding to the operation record file in the source database to determine whether the table structure of the source table has changed, including:

[0074] S201, query the consumption offset record table corresponding to the operation record file in the above source database to obtain the consumption position saved after the last successful consumption of the above operation record file.

[0075] S202, Starting from the aforementioned consumption position, the aforementioned operation log file is parsed to determine whether the table structure of the aforementioned source table has changed.

[0076] In this embodiment of the application, since the data synchronization process may be stopped manually or accidentally, the data synchronization process must start from the last successful consumption position after restarting to ensure that the data is not lost and is not processed repeatedly.

[0077] In one example, the operation log file described above is used to record the types of data operations performed on the source table and information on changes to the table structure.

[0078] By querying the consumption offset record table corresponding to the binlog file of the aforementioned source database, the consumption position saved after the last successful consumption of the aforementioned operation log file can be determined. The operation log file will then be parsed starting from this consumption position.

[0079] Since the operation log file binlog is generated based on the source table that needs to be synchronized, the table structure of the source table is determined based on the data operation type recorded in the binlog to determine whether the table structure has changed, in order to proceed to the next step. For example, if the record in the binlog is a record of a changed table structure, it is determined whether to enable the automatic table structure synchronization function; if the table structure of the source table has not changed, and the operation is a data addition, deletion, or modification, then the data of the source table is synchronized to the target table. In this embodiment, the offset (i.e., consumption position) of the processed binlog data can be saved. The data synchronization processing program processes the binlog of the source database and saves the latest processed consumption position (i.e., offset) of the binlog of the source database.

[0080] In one optional embodiment, the binlog offsets in the source database can be stored in a specific table in the target database and committed within the same transaction as the business table data. (Database transactions are atomic, meaning that all operations within a transaction either succeed or all fail. This operation can include two steps: first, writing data from the upstream database to the downstream database; second, saving the position of the next record in the binlog to the table storing offset information. The success or failure of these two steps must be consistent to prevent data loss or duplicate consumption; therefore, they are committed within a single transaction.) This ensures the consistency of data operations and avoids duplicate data processing or data loss. When the data synchronization process starts, it queries the consumption offset record table storing offset information and continues incrementally consuming binlog data from the last successfully consumed offset, ensuring data accuracy.

[0081] In one example, the table structure of the binlog offset record table can be, but is not limited to, as shown in Table 1 below, including: field name, field type, and field meaning.

[0082] Table 1

[0083]

[0084] In this embodiment, the table structure of the source table is determined based on the operation log file of the source database. If no change has occurred, data synchronization from the source table to the target table is performed. The query and processing process is simple. Therefore, the timeliness of data synchronization between databases can be guaranteed. Furthermore, no manual operation is required, resulting in high security and reduced cost of fixing vulnerabilities.

[0085] In another example, Figure 4 A flowchart illustrating an optional database synchronization method provided in this application embodiment is shown below. Figure 4 As shown, the above method also includes:

[0086] S301, if the table structure of the source table changes and the automatic table structure change function of the target table is enabled, then the table structure of the target table is changed based on the table structure of the source table.

[0087] S302, Update the consumption offset record table corresponding to the above operation log file.

[0088] S303, Starting from the position after the last successful consumption of the above operation log file, parse the above operation log file to determine whether the table structure of the above source table has changed.

[0089] In one example, still as Figure 2As shown, if the table structure of the source table changes, it is determined whether the automatic table structure change function of the target table in the target database is enabled. If the automatic table structure change function is enabled, the table structure of the target table (e.g., with the same table name) in the target database is changed. After completion, the next binlog data is consumed. That is, starting from the next position after the consumption position saved after the last successful consumption of the above operation record file, the next record in the operation record file is parsed to determine whether the table structure of the source table has changed.

[0090] This embodiment of the application can synchronize changes to the table structure of the source database to the target database. When the table structure of the source table changes, the data synchronization processing program captures the change record and then modifies the corresponding target table in the target database to ensure consistency between the source and target table structures. Furthermore, this function can be enabled or disabled as needed. Through the above embodiment, table structure changes can be synchronized in real time, preventing data corruption, facilitating data synchronization, and ensuring data security.

[0091] In one example, Figure 5 A flowchart illustrating an optional database synchronization method provided in this application embodiment is shown below. Figure 5 As shown, the above method also includes:

[0092] S401, if the table structure of the source table changes and the automatic table structure change function of the target table is turned off, then verify the consistency between the table structure of the source table and the table structure of the target table again.

[0093] S402, If the table structure of the source table is inconsistent with the table structure of the target table, wait for a period of time to continue the verification until the number of verifications exceeds the maximum number of retries, then stop the data synchronization processing program and output an error message.

[0094] S403, if the table structure of the source table is consistent with the table structure of the target table, then starting from the next position after the consumption position saved after the last successful consumption of the operation log file, the operation log file is parsed to determine whether the table structure of the source table has changed.

[0095] Still Figure 2 As shown, if the table structure of the source table changes and the automatic table structure change function of the target table is not enabled, the consistency between the table structure of the source table and the table structure of the target table will be checked again.

[0096] In one example, if the table structure of the source table is inconsistent with the table structure of the target table, the process continues to verify after a period of time until the number of verifications exceeds the maximum number of retries. At this point, the data synchronization process stops and an error message is output. In another example, if the table structure of the source table is consistent with the table structure of the target table, the process continues to parse the next record in the operation log file, starting from the position after the last successful consumption of the operation log file, to determine whether the table structure of the source table has changed.

[0097] The above embodiments enable real-time synchronization of table structure changes, preventing data corruption, facilitating data synchronization, and ensuring data security.

[0098] In one optional embodiment, the data synchronization processing method provided in this application has high data security and prevents data corruption, mainly through the following three measures:

[0099] First, after the data synchronization process starts, it will perform table structure verification on the source table in the source database and the target table in the target database. If the table structures are consistent, the data synchronization will continue. If the table structures are inconsistent, it will output error messages and other prompts, and the data synchronization process will stop.

[0100] Second, during the operation of the data synchronization process, the monitoring component mysql-binlog-connector-java is used to monitor the metadata information of the source table and the target table. If the table structure of the source table changes, the table structure of the target table is verified. When the table structures of the source table and the target table are inconsistent, the monitoring component will re-detect after a period of time until the number of verifications exceeds the maximum number of retries. At this point, the data synchronization process stops and displays an error or alarm.

[0101] Third, before adding or modifying data in the target table, the number of fields in the target table and the number of fields in the binlog data of the source table are verified. If they are inconsistent, an error message is printed and the data synchronization process stops.

[0102] In an optional embodiment, the method further includes:

[0103] S501, If ​​the table structure of the source table is inconsistent with the table structure of the target table, the data synchronization process is stopped and an error message is output.

[0104] In this embodiment of the application, as an example, it is still as follows: Figure 2As shown, after the data synchronization process starts, it performs table structure verification on the source table in the source database and the target table in the target database. If the table structures are consistent, data synchronization continues; otherwise, it terminates, outputting error messages and stopping the data synchronization process. This embodiment improves data security during data synchronization and prevents data corruption.

[0105] In another example, after synchronizing the data from the source table to the target table, the method further includes:

[0106] S601, Starting from the position after the last successful consumption of the above operation log file, parse the above operation log file to determine whether the table structure of the above source table has changed.

[0107] In the embodiments of this application, it is still as follows Figure 2 As shown, if the table structure of the source table remains unchanged, the data from the source table will be synchronized to the target table. Then, starting from the offset (consumption position) of the last successful consumption, the binlog data will continue to be consumed incrementally, ensuring the accuracy of the data synchronization process.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0109] According to one or more embodiments of this application, a database synchronization processing apparatus is provided. Figure 6 A structural block diagram of a database synchronization processing device provided in this application embodiment is shown below. Figure 6 As shown, the above-mentioned device includes:

[0110] The acquisition module 601 is used to acquire metadata information of the source table in the source database and metadata information of the target table in the target database.

[0111] The verification module 602 is used to verify the consistency between the table structure of the source table and the table structure of the target table based on the metadata information of the source table and the metadata information of the target table.

[0112] The determination module 603 is used to query the consumption offset record table corresponding to the operation record file of the source database if the table structure of the source table is consistent with the table structure of the target table, so as to determine whether the table structure of the source table has changed.

[0113] The synchronization module 604 is used to synchronize the data of the source table to the target table if the table structure of the source table has not changed.

[0114] In this embodiment, the acquisition module acquires metadata information of the source table in the source database and metadata information of the target table in the target database; the verification module verifies the consistency between the table structure of the source table and the table structure of the target table based on the metadata information of the source table and the target table; the determination module queries the consumption offset record table corresponding to the operation record file of the source database if the table structure of the source table is consistent with the table structure of the target table to determine whether the table structure of the source table has changed; and the synchronization module synchronizes the data of the source table to the target table if the table structure of the source table has not changed.

[0115] In this embodiment, an operation log file from the source database is used to record operation information of the source table, and a consumption offset record table is used to record the consumption address of the last consumption operation log file. Based on the metadata information of the source and target tables, the table structure of the source table is verified to be consistent with that of the target table. If the table structures are consistent, the consumption offset record table corresponding to the operation log file of the source database is queried to determine whether the table structure of the source table has changed. If the table structure of the source table has not changed, the data of the source table is directly synchronized to the target table. This ensures the timeliness of data synchronization between databases, requires no manual operation, has high security, and reduces the cost of fixing vulnerabilities. Therefore, this embodiment can solve the problems of ineffective data synchronization and excessive cost in the prior art.

[0116] According to one or more embodiments of this application, the above-described determining module includes:

[0117] The query unit is used to query the consumption offset record table corresponding to the operation record file in the source database to obtain the consumption position saved after the last successful consumption of the operation record file. The operation record file is used to record the data operation type and table structure change information of the source table.

[0118] The parsing unit is used to parse the operation log file starting from the aforementioned consumption position to determine whether the table structure of the aforementioned source table has changed.

[0119] According to one or more embodiments of this application, the above-described apparatus further includes:

[0120] The modification module is used to modify the table structure of the target table based on the table structure of the source table if the table structure of the source table changes and the automatic modification function of the target table's table structure is enabled.

[0121] The update module is used to update the consumption offset record table corresponding to the above operation log file;

[0122] The first continuing parsing module is used to parse the operation log file starting from the next position after the last successful consumption position after consuming the above operation log file, in order to determine whether the table structure of the above source table has changed.

[0123] According to one or more embodiments of this application, the above-described apparatus further includes:

[0124] The re-verification module is used to re-verify the consistency between the table structure of the source table and the table structure of the target table if the table structure of the source table changes and the automatic table structure change function of the target table is turned off.

[0125] The first stopping module is used to wait for a period of time to continue verification if the table structure of the source table is inconsistent with the table structure of the target table. The verification continues until the number of verifications exceeds the maximum number of retries, at which point the data synchronization processing program is stopped and an error message is output.

[0126] The second parsing module is used to parse the operation log file starting from the next position after the consumption position saved after the last successful consumption of the operation log file if the table structure of the source table is consistent with the table structure of the target table, so as to determine whether the table structure of the source table has changed.

[0127] According to one or more embodiments of this application, the above-described apparatus further includes:

[0128] The second stop module is used to stop the data synchronization process and output an error message if the table structure of the source table is inconsistent with the table structure of the target table.

[0129] According to one or more embodiments of this application, the above-described apparatus further includes:

[0130] The third parsing module is used to parse the operation log file starting from the next position after the last successful consumption position, in order to determine whether the table structure of the source table has changed.

[0131] According to one or more embodiments of this application, the metadata information of the source table and the metadata information of the target table both include: table field name, field type, primary key, and field order.

[0132] In an exemplary embodiment, this application also provides an electronic device, including: a processor, and a memory connected to the processor;

[0133] The aforementioned memory stores instructions executed by the computer;

[0134] The processor executes computer execution instructions stored in the memory to implement any of the methods described above.

[0135] In an exemplary embodiment, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described above.

[0136] In an exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.

[0137] To implement the above embodiments, this application also provides an electronic device. (See reference...) Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of this application. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, messaging devices, game consoles, medical devices, fitness equipment, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0138] like Figure 7As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0139] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0140] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this application.

[0141] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device or apparatus. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0142] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0143] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0144] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The units described in the embodiments of this application can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0147] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0148] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution apparatus or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0150] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A database synchronization processing method, characterized in that, The method includes: Obtain metadata information of the source table in the source database and metadata information of the target table in the target database; Based on the metadata information of the source table and the metadata information of the target table, verify the consistency between the table structure of the source table and the table structure of the target table; If the table structure of the source table is consistent with the table structure of the target table, then query the consumption offset record table corresponding to the operation record file of the source database to determine whether the table structure of the source table has changed. If the table structure of the source table has not changed, the data of the source table is synchronized to the target table, and the position recorded after successful consumption is saved to the consumption offset record table; wherein, the operation of synchronizing the data to the target table and the operation of saving the position recorded after successful consumption to the consumption offset record table are committed in the same database transaction; If the table structure of the source table changes and the automatic table structure change function of the target table is enabled, then the table structure of the target table is changed based on the table structure of the source table; the consumption offset record table corresponding to the operation record file is updated; starting from the next position after the consumption position saved after the last successful consumption of the operation record file, the operation record file is parsed to determine whether the table structure of the source table has changed. If the table structure of the source table changes and the automatic table structure update function of the target table is disabled, the consistency between the table structure of the source table and the table structure of the target table is checked again. If the table structure of the source table and the table structure of the target table are inconsistent, the check continues for a period of time until the number of checks exceeds the maximum number of retries. Then, the data synchronization processing program is stopped and an error message is output. If the table structure of the source table and the table structure of the target table are consistent, the operation record file is parsed starting from the next position after the last successful consumption position to determine whether the table structure of the source table has changed.

2. The method according to claim 1, characterized in that, Query the consumption offset record table corresponding to the operation record file in the source database to determine whether the table structure of the source table has changed, including: Query the consumption offset record table corresponding to the operation record file in the source database to obtain the consumption position saved after the last successful consumption of the operation record file. The operation record file is used to record the data operation type and table structure change information for updating the source table. Starting from the consumption location, the operation log file is parsed to determine whether the table structure of the source table has changed.

3. The method according to claim 1 or 2, characterized in that, The method further includes: If the table structure of the source table is inconsistent with the table structure of the target table, the data synchronization process will stop and an error message will be output.

4. The method according to claim 1 or 2, characterized in that, After synchronizing the data from the source table to the target table, the method further includes: Starting from the next position after the last successful consumption of the operation log file, the operation log file is parsed to determine whether the table structure of the source table has changed.

5. The method according to claim 1 or 2, characterized in that, The metadata information of both the source table and the target table includes: table field name, field type, primary key, and field order.

6. A database synchronization processing device, characterized in that, The device includes: The acquisition module is used to acquire metadata information of the source table in the source database and metadata information of the target table in the target database. The verification module is used to verify the consistency between the table structure of the source table and the table structure of the target table based on the metadata information of the source table and the metadata information of the target table. The determination module is used to query the consumption offset record table corresponding to the operation record file of the source database if the table structure of the source table is consistent with the table structure of the target table, so as to determine whether the table structure of the source table has changed. The synchronization module is used to synchronize the data of the source table to the target table if the table structure of the source table has not changed, and to save the position recorded after successful consumption to the consumption offset record table; wherein the operation of synchronizing the data to the target table and the operation of saving the position recorded after successful consumption to the consumption offset record table are committed in the same database transaction; The modification module is used to modify the table structure of the target table based on the table structure of the source table if the table structure of the source table changes and the automatic modification function of the table structure of the target table is enabled. The update module is used to update the consumption offset record table corresponding to the operation record file; The first continuing parsing module is used to parse the operation log file starting from the next position after the consumption position saved after the last successful consumption of the operation log file, in order to determine whether the table structure of the source table has changed; If the table structure of the source table changes and the automatic table structure change function of the target table is turned off, the consistency between the table structure of the source table and the table structure of the target table will be verified again. The first stopping module is used to wait for a period of time to continue verification if the table structure of the source table is inconsistent with the table structure of the target table, until the number of verifications exceeds the maximum number of retries, then stop the data synchronization processing program and output an error message. The second continuing parsing module is used to parse the operation record file starting from the next position after the consumption position saved after the last successful consumption of the operation record file if the table structure of the source table is consistent with the table structure of the target table, so as to determine whether the table structure of the source table has changed.

7. An electronic device, characterized in that, include: A processor, and a memory connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

Images