Incremental data synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of databases, and discloses an incremental data synchronization method and device, computer equipment and a storage medium. Obtaining a source end DDL statement corresponding to the incremental data needing to be synchronized to the target end database from the source end database; analyzing the source end DDL statement to obtain operation information which is contained in the source end DDL statement and aims at the source end database, and sending the operation information to an intermediate database; consuming the operation information stored in the intermediate database, and performing message processing to assemble a target end DDL statement executable by the target end database; executing the target end DDL statement in the target end database so as to synchronize the incremental data to the target end database; therefore, the incremental data synchronization between the target end database and the source end database can be effectively realized.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to an incremental data synchronization method, device, computer equipment and computer-readable storage medium. Background Art

[0002] At present, with the rapid development of information technology, databases have become the core tools for information management and data storage in modern society. The application of database technology is widespread in many industries such as communications, commerce, and medical care, and they play a vital role in data management and analysis. At present, with the continuous deepening of digital transformation, the demand for databases in various industries is growing, which has promoted the continuous innovation and development of database technology.

[0003] However, there are many database products on the market, each with its own unique design concept, architecture and implementation. These database products include but are not limited to relational databases such as MySQL, PostgreSQL, and Oracle, as well as non-relational databases such as MongoDB and Cassandra. Each database system has its own specific data definition language (DDL), which can perform a series of operations on objects in the database, such as creating, modifying and deleting data tables. Therefore, in the process of synchronizing incremental data between the source database and the target database, the DDL statements executed in the source database need to be accurately synchronized to the target database to ensure the data consistency between the target database and the source database. However, due to the differences in the definition of grammatical rules of database products from different manufacturers, direct synchronization of incremental data between different databases will lead to compatibility issues, affecting the synchronization and sharing of incremental data between the source database and the target database, limiting the liquidity and availability of incremental data, and increasing the cost and risk of enterprises in the process of incremental data synchronization.

[0004] Based on this, how to provide an incremental data synchronization method, device, computer equipment and computer-readable storage medium that can effectively synchronize incremental data between the target database and the source database and ensure the consistency of the data between the two is a problem that urgently needs to be solved by technical personnel in this field. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an incremental data synchronization method, apparatus, computer device and computer-readable storage medium, aiming to solve the problem of how to effectively synchronize incremental data between a target database and a source database to ensure the consistency of data between the two.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides an incremental data synchronization method, which includes:

[0008] In response to triggering a synchronization task of incremental data between a source database and a target database, obtaining, in the source database, a source DDL statement corresponding to the incremental data that needs to be synchronized to the target database;

[0009] Parsing the source-side DDL statement to obtain operation information for the source-side database contained in the source-side DDL statement, and sending the operation information to the intermediate database;

[0010] Consuming the operation information stored in the intermediate database and assembling it into a target-side DDL statement executable by the target-side database after message processing;

[0011] The target-side DDL statement is executed in the target-side database to synchronize the incremental data to the target-side database.

[0012] In a second aspect, the present invention provides an incremental data synchronization device, comprising:

[0013] An acquisition module, configured to, in response to a task of triggering synchronization of incremental data between a source database and a target database, acquire, in the source database, source DDL statements corresponding to the incremental data to be synchronized to the target database;

[0014] A parsing module, used to parse the source-side DDL statement, obtain operation information for the source-side database contained in the source-side DDL statement, and send the operation information to the intermediate database;

[0015] A consumption module, used for consuming the operation information stored in the intermediate database and assembling it into a target-side DDL statement executable by the target-side database after message processing;

[0016] An execution module is used to execute the target-side DDL statement in the target-side database to synchronize the incremental data to the target-side database.

[0017] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the incremental data synchronization method as described above when executing the computer program.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the incremental data synchronization method as described above.

[0019] Compared with the prior art, the present invention provides an incremental data synchronization method, apparatus, computer device and computer-readable storage medium, wherein, in response to a synchronization task that triggers incremental data between a source database and a target database, a source DDL statement corresponding to the incremental data that needs to be synchronized to the target database is obtained in the source database; the source DDL statement is parsed to obtain operation information for the source database contained in the source DDL statement, and the operation information is sent to an intermediate database; the operation information stored in the intermediate database is consumed, and assembled into a target DDL statement executable by the target database after message processing; the target DDL statement is executed in the target database to synchronize the incremental data to the target database; thus, the present invention can effectively realize the synchronization of incremental data between the target database and the source database, and ensure the consistency of data between the two. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of an application environment of an incremental data synchronization method provided in an embodiment of the present invention.

[0022] Figure 2 A flowchart of an incremental data synchronization method provided by an embodiment of the present invention.

[0023] Figure 3 A schematic diagram of a program module of an incremental data synchronization device provided in one embodiment of the present invention.

[0024] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention.

[0025] Figure 5 Another structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0028] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0030] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0032] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0033] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0034] An incremental data synchronization method provided by an embodiment of the present invention can be applied in Figure 1 In the application environment shown, the client and the server communicate through the network. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0035] See also Figure 2 An embodiment of the present invention provides an incremental data synchronization method, wherein the method comprises the following steps:

[0036] S100, in response to triggering a synchronization task of incremental data between a source database and a target database, obtaining, in the source database, a source DDL statement corresponding to the incremental data to be synchronized to the target database;

[0037] S200, parsing the source-side DDL statement to obtain operation information for the source-side database contained in the source-side DDL statement, and sending the operation information to the intermediate database;

[0038] S300, consuming the operation information stored in the intermediate database, and assembling it into a target-side DDL statement executable by the target-side database after message processing;

[0039] S400: Execute the target-side DDL statement in the target-side database to synchronize the incremental data to the target-side database.

[0040] In specific implementation, this embodiment effectively realizes the synchronization of incremental data between the source database and the target database through a series of carefully designed steps, and ensures the consistency of the data of the two. First, in step S100, the system responds to the triggering of the synchronization task, captures the source DDL statements corresponding to the incremental data from the source database, and ensures the real-time tracking and collection of the incremental data. Then, in step S200, by parsing the source DDL statements, specific operation information is extracted, and the operation information is sent to the intermediate database, which provides a basis for subsequent processing. In step S300, the system consumes the operation information stored in the intermediate database, and after message processing, sends these operation information to the intermediate database. The target-side DDL statements are converted into executable target-side DDL statements for the target-side database. This step solves the heterogeneity problem that may exist between the source-side database and the target-side database. Finally, in step S400, the target-side DDL statements are executed in the target-side database to synchronize the incremental data to the target-side database, thereby achieving real-time data update and structural consistency. The entire process of this implementation method ensures the consistency of incremental data between the source-side database and the target-side database through precise DDL statement parsing, conversion and execution. At the same time, through the intervention of an intermediate database, a buffering and conversion mechanism is provided, so that efficient data synchronization can be achieved even in the case of different database types.

[0041] Further, in one embodiment, the incremental data synchronization method, wherein the step S100, in response to triggering the synchronization task of incremental data between the source database and the target database, obtains in the source database a source DDL statement corresponding to the incremental data to be synchronized to the target database, specifically comprises the steps of:

[0042] In response to a synchronization task of incremental data between a source database and a target database triggered by a task-driven condition, the database log of the source database is parsed to obtain the source DDL statement corresponding to the incremental data that needs to be synchronized to the target database; wherein the task-driven condition includes a time interval, a specific event, and manual control.

[0043] In specific implementation, this embodiment realizes real-time capture and processing of source database changes by responding to synchronization tasks triggered by task-driven conditions; specifically, this step uses technical means to parse the source database log, and can accurately extract the source DDL statements corresponding to the incremental data that needs to be synchronized to the target database according to trigger conditions such as time intervals, specific events or manual control. Such technical implementation not only improves the timeliness and accuracy of incremental data synchronization, but also can flexibly respond to different synchronization requirements, ensuring the consistency of data and structure between the source database and the target database, thereby providing a solid foundation for subsequent data consistency maintenance and business continuity.

[0044] The specific implementation process of the steps in this embodiment is roughly as follows:

[0045] 1. Monitoring task driving conditions

[0046] Set up a monitoring mechanism: Set up a monitoring mechanism in the system to detect whether the task-driving conditions are met; including preset time intervals (for example, triggering once every 15 minutes), specific events in the database (such as table creation or modification), and manual triggering instructions from administrators;

[0047] 2. Trigger synchronization tasks

[0048] Task triggering: Once the task driving conditions are detected to be met, the system will automatically trigger the synchronization task of incremental data. This triggering can be automatic or manually initiated by the administrator through the user interface.

[0049] 3. Analyze the source database log

[0050] Log acquisition: The system connects to the source database and obtains database logs; these logs contain all DDL statements that have changed the structure of the source database since the last synchronization;

[0051] Log parsing: The system parses these logs and identifies the source DDL statements corresponding to the incremental data. The parsing process may involve operations such as identifying the log format and extracting DDL statements.

[0052] 4. Extract DDL statements

[0053] DDL statement extraction: From the parsed log data, the system extracts specific source-side DDL statements, which describe changes in the source-side database structure, such as creating a new table, modifying the table structure, and deleting a table.

[0054] Verify DDL statements (optional): Before sending source DDL statements to the intermediate database, the system may verify them to ensure the correctness and completeness of the statements and whether they are applicable to the target database.

[0055] Through the above process, this embodiment realizes real-time monitoring and capturing of changes in the source database structure, laying a foundation for maintaining the consistency of the source database and the target database.

[0056] Further, in one embodiment, the incremental data synchronization method, wherein the source database is a distributed database including a plurality of data nodes;

[0057] The step S200, parsing the source-side DDL statement to obtain operation information for the source-side database contained in the source-side DDL statement, and sending the operation information to the intermediate database, specifically includes the following steps:

[0058] The source-side DDL statement is parsed to obtain operation information for each of the data nodes contained in the source-side DDL statement, and the operation information is sent to the intermediate database for sequential storage.

[0059] In specific implementation, this embodiment can extract specific operation information from multiple data nodes of a distributed database by parsing source-side DDL statements, and store the operation information in an intermediate database in an orderly manner. The implementation of this technology ensures that even in a complex distributed database environment, cross-node data structure changes can be accurately captured and transmitted, thereby providing detailed operation records for subsequent target-side database synchronization, ensuring the integrity and consistency of data synchronization, and also supporting the sequential execution of cross-node operations, which is crucial for maintaining data integrity and performing correct data recovery.

[0060] The specific implementation process of the steps in this embodiment is roughly as follows:

[0061] 1. Parsing DDL statements

[0062] Parse the source DDL statements and extract the operation information for each data node in the source DDL statements;

[0063] 2. Operation information integration

[0064] Integrate the extracted operation information into a unified format for subsequent processing and transmission; this may include encapsulating the information into a specific data structure or message format, such as a JSON object;

[0065] 3. Store in order to the intermediate database

[0066] The integrated operation information is sent to the intermediate database through the network. In order to ensure the order of the operation, the sending process needs to maintain the order of the operation information to ensure that the intermediate database can process the information in the correct order.

[0067] In an intermediate database, store the operations in the order they were received; this may involve writing to a specific queue or table while ensuring data persistence and consistency;

[0068] 4. Confirmation and error handling (optional)

[0069] After the operation information is successfully stored, the intermediate database will send a confirmation signal back to the source database, indicating that the information has been successfully received and stored;

[0070] If an error occurs during parsing or storage, implement error handling mechanisms such as retrying, recording error logs, and notifying administrators to ensure that the problem can be discovered and resolved in a timely manner.

[0071] Through the above process, this embodiment not only realizes the parsing and integration of DDL statement changes of multiple data nodes in a distributed database, but also ensures that these changes can be correctly transmitted and stored in the intermediate database, providing a solid foundation for subsequent data synchronization and consistency maintenance.

[0072] Furthermore, in one embodiment, the incremental data synchronization method, wherein the step S300, consuming the operation information stored in the intermediate database and assembling it into a target-side DDL statement executable by the target-side database after message processing, specifically comprises the following steps:

[0073] The operation information stored in the intermediate database is consumed in sequence, and assembled into a target-side DDL statement executable by the target-side database after parsing, conversion and verification.

[0074] In specific implementation, this embodiment achieves the technical effect of converting the operation information of the source database into the target-side DDL statements executable by the target database by sequentially consuming the operation information stored in the intermediate database and after parsing, conversion and verification. This process ensures that the data structure changes across heterogeneous database environments can be correctly understood and executed, maintains the integrity and consistency of the data, and improves the accuracy and reliability of data synchronization, so that the incremental data synchronization between the source database and the target database can be carried out smoothly.

[0075] The specific implementation process of the steps in this embodiment is roughly as follows:

[0076] 1. Read operation information in sequence

[0077] First, a connection is established with the intermediate database to access the stored operational information;

[0078] Read the operation information from the intermediate database in the order in which it is stored; this ensures the order of operations, which is essential for maintaining data consistency;

[0079] 2. Analyze operation information

[0080] Parse the read operation information to understand its meaning and the required database operation type (such as creating a table, modifying the table structure, etc.);

[0081] 3. Conversion operation information

[0082] According to the syntax and structure of the target database, the parsed operation information is converted into DDL statements that the target database can understand; this may include syntax adjustment, data type mapping, etc.

[0083] 4. Verify DDL statements

[0084] Before execution, verify whether the generated DDL statements are correct and comply with the constraints and rules of the target database to ensure that no errors will occur after execution.

[0085] 5. Assemble the target DDL statement

[0086] Assemble the validated DDL statements into a format that can be executed on the target database; this may involve combining individual DDL statements into batches or scripts for more efficient execution;

[0087] 6. Prepare for execution (optional)

[0088] Record the consumption of operation information and the generation process of DDL statements on the target side to facilitate problem tracking and system auditing;

[0089] Prepare error handling logic to enable rollback or retry when DDL statements fail to execute on the target side;

[0090] Through the above process, this embodiment ensures that the operation information read from the intermediate database can be correctly processed and converted into target-side DDL statements executable by the target-side database, thereby preparing for the next step of data synchronization, namely, executing the target-side DDL statements in the target-side database.

[0091] Further, in one embodiment, the incremental data synchronization method, wherein the step S400, executing the target-side DDL statement in the target-side database to synchronize the incremental data to the target-side database, specifically comprises the steps of:

[0092] Execute the target-side DDL statement in the target-side database, and confirm the execution status of the target-side DDL statement;

[0093] When the execution status of the target-side DDL statement is successful, the incremental data is synchronized from the source-side database to the target-side database.

[0094] Furthermore, the incremental data synchronization method, wherein, after executing the target-side DDL statement in the target-side database and confirming the execution status of the target-side DDL statement, specifically further comprises the steps of:

[0095] When the execution status of the target-side DDL statement is execution failure, error diagnosis is performed;

[0096] When the diagnosis result satisfies the preset retry strategy, the target-side DDL statement is re-executed in the target-side database to synchronize the incremental data from the source-side database to the target-side database.

[0097] In specific implementation, this embodiment implements effective management of execution problems that may occur during incremental data synchronization by executing the converted target-side DDL statements in the target-side database and monitoring their execution status; when the target-side DDL statement is successfully executed, incremental data synchronization can proceed smoothly; if the execution fails, the system can automatically retry to execute the target-side DDL statement until it succeeds through error diagnosis and preset retry strategies, thereby ensuring the final consistency of incremental data and improving the robustness of the synchronization process. The implementation of this technology not only ensures the accurate transmission of data structure changes, but also optimizes the efficiency and reliability of data synchronization through automated error handling and retry mechanisms.

[0098] The specific implementation process of the steps in this embodiment is roughly as follows:

[0099] 1. Execute DDL statements on the target side

[0100] Execute DDL statements: Execute the converted target DDL statements in the target database; this usually involves running these statements directly in the database management system of the target database to apply the structural changes;

[0101] 2. Confirm the execution status

[0102] Check the execution result: After executing the target-side DDL statement, check its execution status immediately; this can be done by querying the job or transaction log of the target-side database to determine whether the target-side DDL statement has been successfully applied;

[0103] 3. Successfully synchronize incremental data

[0104] Synchronize incremental data: If the target DDL statement is executed successfully, continue to perform incremental data synchronization operations; this may involve transferring incremental data captured by the change data capture (CDC) system of the source database to the target database;

[0105] 4. Error diagnosis and retry strategy

[0106] Error diagnosis: If the target DDL statement fails to execute, perform error diagnosis to determine the cause of the failure; this may include checking the error log, analyzing the syntax and semantics of the target DDL statement, and verifying the permissions and configuration of the target database;

[0107] Evaluate retry strategy: Based on the error diagnosis results, evaluate whether the preset retry strategy is met; the retry strategy may include the retry limit, retry interval, and specific error types. These strategies define when the execution of the target-side DDL statement should be retried;

[0108] 5. Retry executing the DDL statement

[0109] Retry DDL execution: If the diagnostic results indicate that the error is temporary or recoverable and meets the retry policy, re-execute the target DDL statement; this process may need to record the number of retries to avoid infinite loops and provide feedback to the system administrator when necessary;

[0110] Monitoring and recording: During the entire process, monitor the execution and retry of DDL statements on the target side and record all related activities to facilitate problem tracking and system auditing;

[0111] Through the above process, this embodiment ensures that the target database can correctly execute the target DDL statement, and can automatically diagnose and retry errors when an execution failure occurs, until the target DDL statement is successfully executed, thereby achieving synchronization of incremental data; this process improves the automation and reliability of data synchronization, and ensures data consistency between the source database and the target database.

[0112] Furthermore, in one embodiment, in the incremental data synchronization method, the source-end database and the target-end database are heterogeneous databases of different database types.

[0113] In specific implementation, this embodiment can realize incremental data synchronization between heterogeneous databases when the source database and the target database are of different database types; wherein, by parsing the source-side DDL statement, converting it into a format suitable for the target-side database, and executing the corresponding target-side DDL statement in the target-side database, it is ensured that even if there may be differences in the database structure and query language, the consistency and integrity of the incremental data can be maintained, thereby realizing efficient data synchronization and accurate transmission of structural changes in a heterogeneous database environment.

[0114] It can be seen from the above method embodiments that the incremental data synchronization method provided by the present invention includes: in response to the synchronization task of the incremental data between the source database and the target database, obtaining the source DDL statement corresponding to the incremental data that needs to be synchronized to the target database in the source database; parsing the source DDL statement to obtain the operation information for the source database contained in the source DDL statement, and sending the operation information to the intermediate database; consuming the operation information stored in the intermediate database, and assembling it into a target DDL statement executable by the target database after message processing; executing the target DDL statement in the target database to synchronize the incremental data to the target database. In this way, the method of the present invention can effectively realize the synchronization of incremental data between the target database and the source database, and ensure the consistency of the data of the two.

[0115] Specifically, the incremental data synchronization method provided by the embodiment of the present invention has at least the following beneficial effects compared with the prior art:

[0116] 1. Compatibility: The method of the present invention does not depend on the characteristics of the database itself, and can be applied to different database types (Oracle, Mysql, DB2), different database architectures (centralized, distributed), and has good compatibility;

[0117] 2. Accuracy: The method of the present invention can accurately synchronize DDL statements and ensure the final consistency of incremental data synchronization between databases;

[0118] 3. Reduce operation and maintenance costs: The method of the present invention can avoid the impact of downstream data synchronization restrictions caused by upstream table structure changes during incremental synchronization, reducing time and manpower operation and maintenance costs.

[0119] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0120] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides an incremental data synchronization device, wherein the device comprises:

[0121] An acquisition module 11 is used for acquiring, in response to a synchronization task of incremental data between a source database and a target database, a source DDL statement corresponding to the incremental data to be synchronized to the target database in the source database;

[0122] A parsing module 12 is used to parse the source-side DDL statement, obtain the operation information for the source-side database contained in the source-side DDL statement, and send the operation information to the intermediate database;

[0123] A consumption module 13, used for consuming the operation information stored in the intermediate database and assembling it into a target-side DDL statement executable by the target-side database after message processing;

[0124] The execution module 14 is used to execute the target-side DDL statement in the target-side database to synchronize the incremental data to the target-side database.

[0125] Furthermore, in one embodiment, in the incremental data synchronization device, the acquisition module 11 is specifically used to:

[0126] In response to a synchronization task of incremental data between a source database and a target database triggered by a task-driven condition, the database log of the source database is parsed to obtain the source DDL statement corresponding to the incremental data that needs to be synchronized to the target database; wherein the task-driven condition includes a time interval, a specific event, and manual control.

[0127] Further, in one embodiment, the incremental data synchronization device, wherein the source database is a distributed database including a plurality of data nodes;

[0128] The analysis module 12 is specifically used for:

[0129] The source-side DDL statement is parsed to obtain operation information for each of the data nodes contained in the source-side DDL statement, and the operation information is sent to the intermediate database for sequential storage.

[0130] Furthermore, in one embodiment, in the incremental data synchronization device, the consumption module 13 is specifically used for:

[0131] The operation information stored in the intermediate database is consumed in sequence, and assembled into a target-side DDL statement executable by the target-side database after parsing, conversion and verification.

[0132] Furthermore, in one embodiment, in the incremental data synchronization device, the execution module 14 is specifically used to:

[0133] Execute the target-side DDL statement in the target-side database, and confirm the execution status of the target-side DDL statement;

[0134] When the execution status of the target-side DDL statement is successful, the incremental data is synchronized from the source-side database to the target-side database.

[0135] Furthermore, the incremental data synchronization device, wherein, after executing the target-side DDL statement in the target-side database and confirming the execution status of the target-side DDL statement, further comprises:

[0136] When the execution status of the target-side DDL statement is execution failure, error diagnosis is performed;

[0137] When the diagnosis result satisfies the preset retry strategy, the target-side DDL statement is re-executed in the target-side database to synchronize the incremental data from the source-side database to the target-side database.

[0138] Furthermore, in one embodiment, in the incremental data synchronization device, the source-end database and the target-end database are heterogeneous databases of different database types.

[0139] It should be noted that in the embodiment of the device of the present invention, the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the aforementioned method embodiment part and will not be repeated here.

[0140] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the service side of the incremental data synchronization method in any of the above method embodiments are implemented.

[0141] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps of the client side of the incremental data synchronization method in any of the above method embodiments are implemented.

[0142] Those skilled in the art will understand that Figure 4 and Figure 5 The structural schematic diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0144] Among them, the memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of a computer device, and in other embodiments, it can also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory can also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory can also be used to temporarily store data that has been output or is to be output.

[0145] Based on the above method embodiments, another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the incremental data synchronization method in any of the above method embodiments is implemented. The computer-readable storage medium may be non-volatile or volatile.

[0146] It should be noted that the above-mentioned functions or steps that can be implemented by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be found in the relevant descriptions in the aforementioned method embodiments. To avoid repetition, they will not be described one by one here.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). The disclosed memory components or memories of the operating environments described herein are intended to comprise one or more of these and / or any other suitable types of memory.

[0148] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, in the embodiment of the device of the present invention, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0149] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0150] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for synchronizing incremental data, characterized in that: include: In response to triggering a synchronization task of incremental data between a source database and a target database, obtaining, in the source database, a source DDL statement corresponding to the incremental data that needs to be synchronized to the target database; Parsing the source-side DDL statement to obtain operation information for the source-side database contained in the source-side DDL statement, and sending the operation information to the intermediate database; Consuming the operation information stored in the intermediate database and assembling it into a target-side DDL statement executable by the target-side database after message processing; The target-side DDL statement is executed in the target-side database to synchronize the incremental data to the target-side database.

2. The incremental data synchronization method according to claim 1, characterized in that: In response to triggering the synchronization task of incremental data between the source database and the target database, obtaining, in the source database, a source DDL statement corresponding to the incremental data to be synchronized to the target database, including: In response to a synchronization task of incremental data between a source database and a target database triggered by a task-driven condition, the database log of the source database is parsed to obtain the source DDL statement corresponding to the incremental data that needs to be synchronized to the target database; wherein the task-driven condition includes a time interval, a specific event, and manual control.

3. The incremental data synchronization method according to claim 1, characterized in that: The source database is a distributed database, which includes multiple data nodes; The parsing the source-side DDL statement to obtain operation information for the source-side database contained in the source-side DDL statement, and sending the operation information to the intermediate database includes: The source-side DDL statement is parsed to obtain operation information for each of the data nodes contained in the source-side DDL statement, and the operation information is sent to the intermediate database for sequential storage.

4. The incremental data synchronization method according to claim 3, characterized in that: The consuming the operation information stored in the intermediate database and assembling it into a target-side DDL statement executable by the target-side database after message processing includes: The operation information stored in the intermediate database is consumed in sequence, and assembled into a target-side DDL statement executable by the target-side database after parsing, conversion and verification.

5. The incremental data synchronization method according to claim 1, characterized in that: The executing the target-side DDL statement in the target-side database to synchronize the incremental data to the target-side database includes: Execute the target-side DDL statement in the target-side database, and confirm the execution status of the target-side DDL statement; When the execution status of the target-side DDL statement is successful, the incremental data is synchronized from the source-side database to the target-side database.

6. The incremental data synchronization method according to claim 5, characterized in that: After executing the target-side DDL statement in the target-side database and confirming the execution status of the target-side DDL statement, the method further includes: When the execution status of the target-side DDL statement is execution failure, error diagnosis is performed; When the diagnosis result satisfies the preset retry strategy, the target-side DDL statement is re-executed in the target-side database to synchronize the incremental data from the source-side database to the target-side database.

7. The incremental data synchronization method according to any one of claims 1 to 6, characterized in that: The source database and the target database are heterogeneous databases of different database types.

8. An incremental data synchronization device, characterized in that: include: An acquisition module, configured to, in response to a task of triggering synchronization of incremental data between a source database and a target database, acquire, in the source database, source DDL statements corresponding to the incremental data to be synchronized to the target database; A parsing module, used to parse the source-side DDL statement, obtain operation information for the source-side database contained in the source-side DDL statement, and send the operation information to the intermediate database; A consumption module, used for consuming the operation information stored in the intermediate database and assembling it into a target-side DDL statement executable by the target-side database after message processing; An execution module is used to execute the target-side DDL statement in the target-side database to synchronize the incremental data to the target-side database.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the incremental data synchronization method as described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the incremental data synchronization method as described in any one of claims 1-7 is implemented.

Images