Heterogeneous database structure dynamic synchronization method, equipment and medium

Abstract

The invention discloses a heterogeneous database structure dynamic synchronization method and device and a medium, belongs to the technical field of database migration, and is used for solving the defects of the existing database migration technology in the aspects of environment without native change log support, structure synchronization and fault tolerance. The technical problems of log dependence, low structure repairing efficiency and weak migration fault-tolerant capability exist in the prior art. The method comprises the following steps: performing data structure differentiation comparison between a source library and a target library to generate a structure difference atlas; traversing and converting the abstract syntax tree after syntactic analysis in the source library to generate a DDL statement conforming to the target library; performing incremental synchronization processing on the migration task between the source library and the target library to obtain incremental synchronization information; if structural difference information is detected in the incremental synchronization information, carrying out structural repair processing on the target library; and if the structure repair result information is repair operation failure information, performing data fusing processing on the migration task.

Description

Technical Field

[0001] The present application relates to the technical field of database migration, and in particular to a method, device, and medium for dynamically synchronizing heterogeneous database structures. Background Art

[0002] With the rapid development of information technology, enterprises' data management needs are becoming increasingly complex. Migrating and synchronizing heterogeneous databases has become a critical task. In database migration scenarios, ensuring structural consistency between the source and target databases and ensuring fault tolerance during the migration process are crucial.

[0003] Currently, mainstream database migration tools rely heavily on the database's native change logs to synchronize database structures. For example, MySQL relies on the binlog, and Oracle relies on the redo log. These logs record detailed database structure changes. Migration tools capture this log information to synchronize the target database structure. However, this reliance on native change logs has significant limitations. First, not all databases support native change logs. Some small or specialized databases may not provide such logging functionality due to design architecture or resource constraints. Second, even if a database has native change logs, they may not be enabled in real-world scenarios for various reasons, such as performance considerations or security policy restrictions. In these cases, existing migration tools are unable to automatically capture structural changes, resulting in inability to synchronize database structure, seriously impacting the automation and efficiency of the migration process.

[0004] At the same time, existing solutions typically detect database structural discrepancies passively. They often only passively discover structural discrepancies between the source and target databases after errors occur during data synchronization. This detection method has a significant lag, potentially leading to structural inconsistencies being discovered only after data synchronization has already progressed for some time, or even after most of the data has been synchronized. This not only requires additional time and effort to locate the issue, but may also necessitate reprocessing of the synchronized data due to mismatches with the target structure, significantly increasing migration costs.

[0005] Furthermore, when database structural anomalies are discovered, existing technologies are extremely inefficient in repairing them. This is especially true in cross-database migration scenarios, such as migrating from MySQL to PostgreSQL. Due to significant differences in syntax, data types, and other aspects between the two databases, repairing structural anomalies often requires manually written adaptation scripts. This not only requires in-depth technical knowledge of both databases, but also consumes considerable time and effort to write and debug these scripts. Furthermore, manually written scripts are inevitably prone to errors, further impacting the accuracy and stability of the migration.

[0006] In summary, existing database migration technologies have many shortcomings in environments without native change log support, as well as in structural synchronization and fault tolerance. A new method and system are urgently needed to solve these problems. Summary of the Invention

[0007] The embodiments of the present application provide a method, device, and medium for dynamic synchronization of heterogeneous database structures, which are used to solve the following technical problems: existing database migration technologies have many deficiencies in environments without native change log support, as well as in structural synchronization and fault tolerance, such as log dependency, low structural repair efficiency, and weak migration fault tolerance.

[0008] The embodiments of this application adopt the following technical solutions:

[0009] On the one hand, an embodiment of the present application provides a method for dynamic synchronization of heterogeneous database structures, comprising: performing a data structure difference comparison between a source library and a target library using a preset multimodal structure synchronization engine to generate a structure difference map; wherein the multimodal structure synchronization engine includes an active detection layer and a passive triggering layer; traversing and transforming the parsed abstract syntax tree in the source library according to the initial DDL statement corresponding to the structure difference map, generating a DDL statement that conforms to the target library, and obtaining structural consistency information after the target library executes the structure change; based on the structural consistency information, performing incremental synchronization processing on the migration task between the source library and the target library to obtain incremental synchronization information; if structural difference information is detected in the incremental synchronization information, performing structural repair processing on the target library to obtain structural repair result information; if the structural repair result information indicates that the repair operation failed, performing data fusing processing on the migration task, and performing consistency verification processing between the source library and the target library in the fusing state until re-entering incremental synchronization processing; if the incremental synchronization information is in a completed state, marking the migration task as migration completed information.

[0010] The embodiments of this application detect structural differences through a combination of active detection and passive triggering; a cross-platform DDL converter uses AST parsing to convert DDL statements from different databases; a migration state machine model defines the migration process to ensure orderly migration; and a breakpoint resume and consistency assurance mechanism uses cursor triples, intelligent recovery strategies, and a bidirectional validation protocol to ensure migration fault tolerance and data consistency. This comprehensively addresses the shortcomings of existing technologies in heterogeneous database structure synchronization and migration fault tolerance, improves the automation, efficiency, stability, and reliability of database migration, and provides strong technical support for enterprises performing database migration in complex data environments. It has extremely high practical value and promotional significance.

[0011] In a feasible implementation, a preset multimodal structure synchronization engine is used to perform data structure difference comparison between a source library and a target library to generate a structure difference map, specifically including: based on the active detection layer, and through a database connection module, connecting the source library and the target library to extract detailed metadata information; wherein, the detailed metadata information includes at least: table structure, field definition and index information; transmitting the detailed metadata information in the source library and the detailed metadata information in the target library to a structure difference analysis module; based on the data difference comparison in the structure difference analysis module, generating a first structure difference map under the active detection layer; based on the passive trigger layer, and through an SQL exception parser, performing exception information capture processing on the data synchronization information between the source library and the target library to obtain structural error information; transmitting the structural error information to the structure difference analysis module to generate a second structure difference map under the passive trigger layer; combining the first structure difference map and the second structure difference map to obtain the structure difference map.

[0012] In a feasible implementation, according to the initial DDL statement corresponding to the structural difference graph, the abstract syntax tree after syntax analysis in the source library is traversed and converted to generate a DDL statement that conforms to the target library, and obtain structural consistency information after the target library executes the structural change, specifically including: extracting the structural difference information in the structural difference graph through a cross-platform DDL converter, and generating the initial DDL statement to be executed from the structural difference information; performing syntax analysis on the initial DDL statement in the source library through an AST parsing module to construct the abstract syntax tree; wherein each node in the abstract syntax tree represents a syntax element; according to the abstract syntax tree and through the data type rules of the target library, traversing and converting the abstract syntax tree in the source library to generate a DDL statement that conforms to the target library; inputting the DDL statement into a DDL execution module, and performing a structural change operation on the target library to make the library structure between the target library and the source library consistent, and generating the structural consistency information.

[0013] In a feasible implementation, based on the structural consistency information, the migration task between the source library and the target library is incrementally synchronized to obtain incremental synchronization information, specifically including: when the structural consistency information is detected, starting the migration task; performing a full scan on the structure of the source library, and transmitting the scanned complete structural information to the target library to realize the structural initialization of the target library and determine the initial synchronization status information; if the initial synchronization status information is detected, performing data change monitoring on the source library in an incremental state to determine the incremental change data; wherein, the data change includes at least: data insertion, data update and data parameters; through the preset data change rules, the incremental change data in the source library is synchronized to the target library, and the incremental synchronization information is generated.

[0014] In a feasible implementation, if structural difference information is detected in the incremental synchronization information, the target library is structurally repaired to obtain structural repair result information, specifically including: if during the generation of the incremental synchronization information, the multimodal structural synchronization engine detects that there is the structural difference information between the source library and the target library, a repair DDL statement corresponding to the structural difference information is generated through a cross-platform DDL converter, and the migration task in the incremental synchronization state is converted into a migration task in the structural repair state; data backup is performed on the current data in the target library; based on the repair DDL statement, the structural change operation is re-executed on the target library to generate the structural repair result information; wherein, the structural repair result information includes: repair operation failure information and repair operation success information; the repair operation success information corresponds to continuing to execute incremental synchronization; the repair operation failure information corresponds to executing data fuse.

[0015] In a feasible implementation, if the structural repair result information is repair operation failure information, the migration task is subjected to data fusing processing, and the source library in the fusing state and the target library are subjected to consistency verification processing until the incremental synchronization processing is re-entered, specifically including: if the structural repair result information is repair operation failure information, suspending the data synchronization between the source library and the target library; recording the synchronization error information of the migration task in the fusing state; wherein the synchronization error information includes at least: error type, error occurrence time, error table and error field; through manual intervention, a comprehensive comparison of the data structure between the source library in the fusing state and the target library is performed to obtain consistency verification result information; if the consistency verification result information is inconsistent information, the target library is continued to be structurally repaired until the consistency verification result information is consistent information; if the consistency verification result information is consistent information, the migration task between the source library and the target library is continued to be incrementally synchronized, and incremental synchronization information after consistency verification is generated.

[0016] In a feasible implementation, based on the structural consistency information, incremental synchronization processing is performed on the migration task between the source library and the target library to obtain incremental synchronization information, which also includes: recording the cursor triplet information of the migration task under incremental synchronization; wherein, the cursor triplet information includes: current transaction ID, last successful timestamp and synchronized data fingerprint; the cursor triplet information is synchronously stored in the storage module in real time to realize data reading processing of secondary synchronization of the migration task in the migration interruption state.

[0017] In a feasible implementation manner, if the incremental synchronization information is in a completed state, the migration task is marked as migration completion information, and the method further includes: if it is identified that the migration task is in a migration interruption state, the fault interruption type is identified; wherein, the fault interruption type includes: network failure and hardware failure; if it is the network failure, the reconnection interval is gradually increased through the exponential backoff algorithm, and it is determined to be a waiting for reconnection recovery strategy; if it is the hardware failure, based on the disaster recovery switching rule, the backup environment is switched to continue to execute the migration task, and the validity of the cursor triplet information when the source library is in a breakpoint state is verified; if the cursor triplet information is valid information, according to the interruption duration and the number of changed rows of the source library, and based on the predefined data recovery rules, Automatically select a fault breakpoint recovery strategy; wherein, the fault breakpoint recovery strategy includes: incremental resume mode, differential patch synchronization mode and full verification + incremental retransmission mode; if the cursor triplet information is invalid information, the data synchronization of the migration task is suspended; after the waiting for reconnection recovery strategy and / or the fault breakpoint recovery strategy executes corresponding operations, a two-way verification of the relevant hash values ​​is performed between the source library and the target library to obtain two-way verification result information; if the two-way verification result information is two-way consistent information, the migration task continues to perform incremental synchronization processing; if the two-way verification result information is two-way inconsistent information, the migration task is re-performed with the recovery strategy repair process under the fault interruption state until the migration task continues to perform incremental synchronization processing.

[0018] In a second aspect, an embodiment of the present application also provides a device for dynamic synchronization of heterogeneous database structures, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute a method for dynamic synchronization of heterogeneous database structures described in any of the above embodiments.

[0019] In a third aspect, an embodiment of the present application further provides a non-volatile computer storage medium, which is a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores at least one program, each of which includes instructions. When the instructions are executed by a terminal, the terminal executes a method for dynamic synchronization of heterogeneous database structures as described in any of the above embodiments.

[0020] This application provides a method, device, and medium for dynamically synchronizing heterogeneous database structures. Compared with the prior art, the embodiments of this application have the following beneficial technical effects:

[0021] 1. From the perspective of structural synchronization, the multimodal structural synchronization engine greatly improves the comprehensiveness and timeliness of structural difference detection. The active detection layer can flexibly configure the time window to scan metadata according to the actual business scenario. Regardless of whether the database's native change log is available, it can regularly check for structural differences to avoid the accumulation of a large number of structural inconsistencies due to long-term non-detection. The passive trigger layer responds immediately to SQL anomalies, triggering structural comparison at the moment of data synchronization error. Compared with traditional passive detection, it no longer lags behind data synchronization errors, captures potential structural anomalies in a timely manner, and ensures real-time structural consistency during the data synchronization process. This means that the structural synchronization of heterogeneous databases is no longer limited to the native log function of the database, greatly broadens the scope of application, and greatly improves the compatibility and versatility of the system.

[0022] 2. The cross-platform DDL converter automatically converts DDL statements between different database types based on AST parsing and syntax conversion, effectively resolving syntax incompatibilities during cross-database migrations. For example, using MySQL to PostgreSQL as an example, precise data type mapping and syntax structure adjustments avoid the tedious and error-prone manual adaptation script writing. This not only reduces labor and time costs but also significantly improves the accuracy and automation of structural repairs, reducing the risk of migration failure due to human error and improving migration efficiency and quality.

[0023] 3. The migration state machine model provides orderly control over the entire migration process by clearly defining the migration process and state transition conditions. When encountering complex situations such as structural changes or repair failures, the system can handle them methodically according to the state machine rules, avoiding confusion and errors. For example, a structural repair failure will cause synchronization to suspend in a circuit-breaker state to prevent the spread of errors. After manual intervention, a consistency check will be performed before resuming synchronization. This ensures the stability and reliability of the migration process, enhances the system's ability to respond to various abnormal situations, and reduces migration interruptions and data loss caused by these exceptions.

[0024] 4. Breakpoint resumption and consistency assurance mechanisms provide reliable fault tolerance for the migration process. Cursor triples accurately record synchronization breakpoints, ensuring rapid recovery after a migration interruption. Intelligent recovery strategies automatically select the optimal recovery mode based on the duration of the interruption and the amount of data changes, balancing efficiency and ensuring data integrity. A bidirectional verification protocol further verifies the consistency of synchronized data, allowing for timely detection and repair of inconsistencies during the recovery process. This mechanism significantly improves the system's resilience to unexpected situations such as network and hardware failures, ensuring the continuity of migration tasks and avoiding the need to re-do a full migration due to unexpected interruptions, saving significant time and resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 A flow chart of a method for dynamic synchronization of heterogeneous database structures provided in an embodiment of the present application;

[0027] Figure 2 A main flow chart of structure synchronization and exception handling provided by an embodiment of the present application;

[0028] Figure 3 A flow chart of a multimodal structure synchronization engine provided in an embodiment of the present application;

[0029] Figure 4 A state transition diagram of a migration state machine model provided in an embodiment of the present application;

[0030] Figure 5 A flow chart of a breakpoint resume and consistency assurance mechanism provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the structure of a heterogeneous database structure dynamic synchronization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0033] The embodiment of the present application provides a method for dynamic synchronization of heterogeneous database structures, such as Figure 1 As shown, the method for dynamic synchronization of heterogeneous database structures specifically includes steps S101-S106:

[0034] S101: Using a pre-set multi-modal structure synchronization engine, perform data structure difference comparison between the source database and the target database to generate a structure difference map. The multi-modal structure synchronization engine includes an active detection layer and a passive trigger layer.

[0035] Specifically, based on the active detection layer and through the database connection module, the source database and the target database are connected to extract detailed metadata information, which includes at least table structure, field definition and index information.

[0036] Furthermore, the detailed metadata information in the source library and the detailed metadata information in the target library are both transmitted to the structural difference analysis module.

[0037] Furthermore, based on the data differential comparison in the structural difference analysis module, a first structural difference map under the active detection layer is generated.

[0038] In one embodiment, Figure 2 A main flow chart of structure synchronization and exception handling provided by the embodiment of the present application is as follows: Figure 2 As shown, the system will start the scanning task according to the set configurable time window (1min-24h). First, connect to the source library and target library respectively through the database connection module, and use the metadata query interface provided by the database (such as MySQL's information_schema and PostgreSQL's pg_catalog) to extract detailed metadata information of the source library and target library. This information includes table structure, field definition, index information, etc. Next, the extracted source and target library metadata are transmitted to the structural difference analysis module, which will perform a detailed comparison of the two and generate a structural difference map. The map shows the differences between the source and target libraries in terms of tables, fields, indexes, etc. in an intuitive form, providing clear guidance for subsequent structural synchronization. For example, if a new field is added to a table in the source library, and the field does not exist in the target library, the structural difference map will clearly identify the table and the specific field differences.

[0039] Furthermore, based on the passive trigger layer and through the SQL exception parser, the data synchronization information between the source database and the target database is captured and processed to obtain structural error information.

[0040] Furthermore, the structural error information is transmitted to the structural difference analysis module to generate a second structural difference map under the passive trigger layer.

[0041] Furthermore, the first structural difference map and the second structural difference map are combined to obtain a structural difference map.

[0042] In one embodiment, Figure 2As shown in the figure, during data synchronization, the SQL exception parser monitors exception information generated by data synchronization operations in real time. Upon detecting a structural error, such as a "column not exist" error, the SQL exception parser immediately passes the exception information to the difference comparison trigger module. The structural difference analysis module then initiates the structural difference comparison process. Similar to the active detection layer, it extracts the latest metadata from the source and target libraries and compares them to identify the specific structural differences that caused the exception.

[0043] As a feasible implementation method, Figure 3 A multi-modal structure synchronization engine flow chart provided in the embodiment of the present application is as follows: Figure 3 As shown, the overall structure is divided into two main branches: the active detection layer and the passive triggering layer. The active detection layer begins with a timer trigger, then connects to the source and target libraries to extract metadata. After structural difference analysis, it determines whether any differences exist. If so, a DDL patch is generated; otherwise, a check log is recorded. The passive triggering layer begins with a data synchronization anomaly. After error type analysis, a snapshot of the abnormal table structure is extracted. A structural difference comparison is then performed to generate an emergency repair script. Finally, the DDL patch generated by the active detection layer and the emergency repair script generated by the passive triggering layer are entered into the DDL execution queue to execute the structural change.

[0044] S102. According to the initial DDL statement corresponding to the structural difference graph, the abstract syntax tree after syntax analysis in the source database is traversed and converted to generate DDL statements that conform to the target database, and obtain structural consistency information after the target database executes the structural change.

[0045] Specifically, a cross-platform DDL converter is used to extract structural difference information from the structural difference graph, and the structural difference information is used to generate initial DDL statements to be executed.

[0046] Furthermore, the AST parsing module performs syntax analysis on the initial DDL statements in the source library to construct an abstract syntax tree, in which each node represents a syntax element.

[0047] Furthermore, based on the abstract syntax tree and the data type rules of the target library, the abstract syntax tree in the source library is traversed and converted to generate DDL statements that conform to the target library.

[0048] Furthermore, the DDL statement is input into the DDL execution module, and a structure change operation is performed on the target database to keep the database structure between the target database and the source database consistent, thereby generating structure consistency information.

[0049] In one embodiment, using a cross-platform DDL converter, when the multimodal structure synchronization engine detects structural differences and generates DDL statements that need to be executed, these statements will first enter the AST parsing module. This module will perform syntax analysis on the DDL statements of the source library and build an abstract syntax tree. During the construction process, the DDL statements will be decomposed into individual syntax nodes, each representing a syntax element. For example, the table name and field definition in the table creation statement will become independent nodes. The nodes form a tree structure through parent-child and sibling relationships, accurately reflecting the syntax logic of the DDL statements.

[0050] In one embodiment, the syntax conversion module traverses and converts the abstract syntax tree according to the target database type rules. For example, when migrating from MySQL to PostgreSQL, nodes related to data types are replaced based on predefined type mapping tables (e.g., "INT(11)":"INTEGER", "VARCHAR(255)":"TEXT", "DATETIME":"TIMESTAMP WITH TIME ZONE"). Furthermore, adjustments are made to address differences in syntax structure between different databases, such as statement keywords and symbols.

[0051] In one embodiment, after the conversion is completed, DDL statements that conform to the target database syntax are generated. These statements will be sent to the DDL execution module to execute structure change operations in the target database to ensure that the target database structure remains consistent with the source database.

[0052] S103: Based on the structural consistency information, incremental synchronization processing is performed on the migration task between the source database and the target database to obtain incremental synchronization information.

[0053] Specifically, when structural consistency information is detected, the migration task is started.

[0054] Furthermore, a full scan is performed on the structure of the source library, and the scanned complete structure information is transmitted to the target library to initialize the structure of the target library and determine the initial synchronization state information.

[0055] If the initial synchronization status information is detected, the source database is monitored for data changes in the incremental state to determine the incremental change data. Data changes include at least: data insertion, data update, and data parameters.

[0056] Furthermore, the incremental change data in the source database is synchronized to the target database through the preset data change rules, and incremental synchronization information is generated.

[0057] In one embodiment, after the migration task is started, it first enters the initial synchronization state. During this stage, the system will perform a comprehensive scan of the source database's structure and transfer the complete structural information to the target database, completing the structural initialization of the target database. At the same time, the system records the relevant information of the initial synchronization to prepare for subsequent incremental synchronization. After the initial synchronization (initial synchronization status information) is completed, the system enters the incremental synchronization state. In this state, the system continuously monitors data changes in the source database, including data insertion, update, and deletion operations. When changes in the source database are captured, the changed data will be synchronized to the target database according to pre-set rules.

[0058] As a feasible implementation, cursor triplet information for migration tasks undergoing incremental synchronization is recorded. This cursor triplet information includes the current transaction ID, the last success timestamp, and the synchronized data fingerprint. This cursor triplet information is synchronously stored in a storage module in real time to facilitate data read processing for secondary synchronization of migration tasks during an interrupted migration state.

[0059] In one embodiment, during incremental synchronization, the system continuously records cursor triples (current transaction ID, last success timestamp, and synchronized data fingerprint). This information is stored in a dedicated storage module, such as a relational database or distributed cache, to facilitate rapid access after a migration interruption. For example, each time a transaction is successfully synchronized, the current transaction ID and last success timestamp are updated, and the synchronized data fingerprint is calculated and stored.

[0060] S104: If structural difference information is detected in the incremental synchronization information, a structural repair process is performed on the target library to obtain structural repair result information.

[0061] Specifically, if, during the generation of incremental synchronization information, the multimodal structure synchronization engine detects that there is structural difference information between the source and target libraries, a cross-platform DDL converter generates a repair DDL statement corresponding to the structural difference information, and converts the migration task in the incremental synchronization state into a migration task in the structure repair state.

[0062] Furthermore, the current data in the target database is backed up.

[0063] Furthermore, based on the repair DDL statement, the structure change operation is re-executed on the target database to generate structure repair result information. This structure repair result information includes repair operation failure information and repair operation success information. Successful repair operation information indicates that incremental synchronization will continue. Failed repair operation information indicates that data circuit breaking will be executed.

[0064] In one embodiment, Figure 2As shown in the figure, during incremental synchronization, if the multimodal structure synchronization engine detects structural differences, the system will transition from incremental synchronization to structure repair. At this point, the cross-platform DDL converter generates the DDL statements required for repair (repair DDL statements) and executes the structural changes in the target database. Before executing the changes, the target database data is backed up to prevent data loss caused by errors during the change process.

[0065] S105: If the structure repair result information indicates that the repair operation has failed, data fusing is performed on the migration task, and consistency check is performed between the source database and the target database in the fusing state until the incremental synchronization process is re-entered.

[0066] Specifically, if the structure repair result information is repair operation failure information, the data synchronization between the source database and the target database is suspended.

[0067] Furthermore, synchronization error information when the migration task is in a fuse state is recorded, wherein the synchronization error information at least includes: error type, error occurrence time, error table, and error field.

[0068] Furthermore, through manual intervention, a comprehensive comparison of the data structures between the source database and the target database in the blown state is performed to obtain consistency verification result information.

[0069] Furthermore, if the consistency check result information is inconsistent, the target database will continue to perform structural repair processing until the consistency check result information is consistent. If the consistency check result information is consistent, the migration task between the source database and the target database will continue to be incrementally synchronized, and the incremental synchronization information after consistency verification will be generated.

[0070] In one embodiment, Figure 2 As shown, if the structure repair operation fails, the system will enter the circuit breaker state. In the circuit breaker state, data synchronization operations will be suspended to prevent further escalation of the error. Detailed error information, including the error type, occurrence time, affected tables and fields, will be recorded to facilitate manual investigation and resolution. In the circuit breaker state, after manual intervention, the system enters the consistency verification phase. This phase conducts a comprehensive comparison of the source and target database data to ensure data and structural consistency. If inconsistencies are found, the structure repair or data synchronization operation will be performed again until the data is fully consistent. After the consistency verification is complete, the system will return to the incremental synchronization state and continue the data migration.

[0071] As a feasible implementation method, Figure 4 A state transition diagram of a migration state machine model provided in an embodiment of the present application is shown as follows: Figure 4As shown, starting from the INITIALIZED state, the system enters the SCHEMA_SYNC state through full structural synchronization. After SCHEMA_SYNC succeeds, it enters the DATA_SYNC state. The DATA_SYNC state is in the RUNNING state while data synchronization is in progress. If a structural error is detected in the RUNNING state, the system enters the SCHEMA_REPAIR state. If the repair is successful, the system returns to the DATA_SYNC state. If the repair fails, the system enters the METADATA_ROLLBACK state.

[0072] If the METADATA_ROLLBACK rollback is successful, the state enters the PAUSED state. After manual intervention, the state returns to the SCHEMA_SYNC state. If the RUNNING state encounters a network interruption, the state enters the RETRYING state. If the retry succeeds, the state returns to the DATA_SYNC state. If the retry timeout occurs, the state enters the FAILED state.

[0073] S106: If the incremental synchronization information is in a completed state, the migration task is marked as migration completion information.

[0074] Specifically, before the incremental synchronization information is in the completed state, if it is identified that the migration task is in the migration interrupted state, the fault interruption type is identified, wherein the fault interruption type includes: network failure and hardware failure.

[0075] Furthermore, if it is a network failure, the reconnection interval is gradually increased through an exponential backoff algorithm, and the waiting for reconnection recovery strategy is determined.

[0076] Furthermore, if it is a hardware failure, based on the disaster recovery switching rules, the backup environment is switched to continue the migration task, and the validity of the downstream triplet information when the source library is in a breakpoint state is verified.

[0077] Furthermore, if the cursor triplet information is valid, a fault recovery strategy is automatically selected based on the interruption duration, the number of changed rows in the source database, and predefined data recovery rules. These fault recovery strategies include incremental resume mode, differential patch synchronization mode, and full verification + incremental retransmission mode. If the cursor triplet information is invalid, data synchronization for the migration task is suspended.

[0078] In one embodiment, when a migration task is interrupted due to a failure, the system will first analyze the cause of the interruption. If it is a network failure, the waiting reconnection mechanism will be activated, and the exponential backoff algorithm will be used to gradually increase the reconnection interval to avoid frequent reconnections causing excessive pressure on the network. If it is a hardware failure, the disaster recovery switching process will be triggered, and the system will switch to the backup hardware environment to continue the migration task. After successfully reconnecting or switching to the disaster recovery environment, the system will read the breakpoint cursor information and verify the validity of the source library cursor. If the cursor is valid, the target library data consistency will be further verified. According to the duration of the interruption and the number of rows changed in the source library, the recovery mode is automatically selected according to predefined rules. If the interruption duration is less than 5 minutes and the number of changed rows in the source database is less than 1,000, the incremental resume mode is selected, that is, the unsynchronized data is synchronized from the breakpoint. If the interruption duration is less than 1 hour and the number of changed rows in the source database is less than 100,000, the differential patch synchronization mode is used. By calculating the data differences between the source and target databases, a differential patch is generated and applied to the target database. If the interruption duration is longer than 1 hour, the full verification + incremental retransmission mode is used. First, the source and target databases are fully verified to find inconsistent data, and then the data is resynchronized.

[0079] Furthermore, after waiting for the reconnection recovery strategy and / or the fault breakpoint recovery strategy to execute corresponding operations, a bidirectional verification of the relevant hash values ​​is performed between the source database and the target database to obtain bidirectional verification result information.

[0080] If the bidirectional verification result is consistent, the migration task continues with incremental synchronization. If the bidirectional verification result is inconsistent, the migration task is re-processed using the recovery strategy for the interrupted state until the migration task continues with incremental synchronization.

[0081] In one embodiment, after selecting a recovery mode (waiting for reconnection recovery strategy and / or fault breakpoint recovery strategy) and executing the corresponding operation, the system will use a two-way verification protocol to re-verify the data in the source and target databases. On the one hand, the system reads the data from the source database and calculates its hash value. On the other hand, the system reads the corresponding data from the target database and calculates the hash value. The two hash values ​​are compared to see if they are consistent. If they are consistent, the data synchronization is successful and the subsequent incremental synchronization operation can continue. If they are inconsistent, the system will re-check and execute the corresponding repair operation to ensure data consistency.

[0082] As a feasible implementation method, Figure 5 A flow chart of a breakpoint resume and consistency guarantee mechanism provided in an embodiment of the present application is as follows: Figure 5As shown, the migration task begins with an interruption. After analyzing the cause of the interruption, if it's a network failure, the process waits for reconnection (exponential backoff). If it's a hardware failure, a disaster recovery switch is triggered. The breakpoint cursor is then read to verify the source database's cursor validity. If valid, the target database's data consistency is verified. If consistent, incremental synchronization continues. If inconsistent, a differential patch is generated, applied, and the cursor updated, continuing incremental synchronization. If the cursor is invalid, the process rolls back to the most recent valid snapshot, triggering a full data verification before continuing incremental synchronization.

[0083] In addition, the embodiment of the present application also provides a heterogeneous database structure dynamic synchronization device, such as Figure 6 As shown, the heterogeneous database structure dynamic synchronization device 600 specifically includes:

[0084] At least one processor 601. And a memory 602 in communication with the at least one processor 601. The memory 602 stores instructions that can be executed by the at least one processor 601, so that the at least one processor 601 can execute:

[0085] Through the preset multimodal structure synchronization engine, the data structure differences between the source library and the target library are compared to generate a structure difference map. The multimodal structure synchronization engine includes: an active detection layer and a passive trigger layer;

[0086] Based on the initial DDL statements corresponding to the structural difference graph, the abstract syntax tree after syntax analysis in the source database is traversed and converted to generate DDL statements that conform to the target database, and the structural consistency information after the target database executes the structural change is obtained;

[0087] Based on the structural consistency information, incremental synchronization is performed on the migration tasks between the source and target databases to obtain incremental synchronization information.

[0088] If structural difference information is detected in the incremental synchronization information, the target database is structurally repaired to obtain structural repair result information;

[0089] If the structure repair result indicates a repair operation failure, the migration task is subjected to data fusing. A consistency check is performed between the source and target databases in the fusing state until incremental synchronization is resumed.

[0090] If the incremental synchronization information is in the completed state, the migration task is marked as migration completed.

[0091] The embodiments of this application detect structural differences through a combination of active detection and passive triggering; a cross-platform DDL converter uses AST parsing to convert DDL statements from different databases; a migration state machine model defines the migration process to ensure orderly migration; and a breakpoint resume and consistency assurance mechanism uses cursor triples, intelligent recovery strategies, and a bidirectional validation protocol to ensure migration fault tolerance and data consistency. This comprehensively addresses the shortcomings of existing technologies in heterogeneous database structure synchronization and migration fault tolerance, improves the automation, efficiency, stability, and reliability of database migration, and provides strong technical support for enterprises performing database migration in complex data environments. It has extremely high practical value and promotional significance.

[0092] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0093] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0094] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0095] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0099] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0100] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0102] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the specification of the present application.

Claims

1. A method for dynamic synchronization of heterogeneous database structures, characterized in that: The method comprises: Through a preset multimodal structure synchronization engine, data structure differences between the source library and the target library are compared to generate a structure difference map; wherein, the multimodal structure synchronization engine includes: an active detection layer and a passive trigger layer; According to the initial DDL statement corresponding to the structural difference graph, the abstract syntax tree after syntax analysis in the source database is traversed and converted to generate DDL statements that conform to the target database, and the structural consistency information after the target database executes the structural change is obtained; Based on the structural consistency information, incremental synchronization processing is performed on the migration task between the source database and the target database to obtain incremental synchronization information; If structural difference information is detected in the incremental synchronization information, a structural repair process is performed on the target library to obtain structural repair result information; If the structure repair result information indicates that the repair operation has failed, data fusing is performed on the migration task, and consistency check is performed between the source database and the target database in the fusing state until incremental synchronization is re-entered. If the incremental synchronization information is in a completed state, the migration task is marked as migration completion information.

2. A method for dynamic synchronization of heterogeneous database structures according to claim 1, characterized in that: Through the preset multimodal structure synchronization engine, the data structure differences between the source and target libraries are compared and a structural difference map is generated, including: Based on the active detection layer and through the database connection module, the source database and the target database are connected to extract detailed metadata information; wherein the detailed metadata information includes at least: table structure, field definition and index information; Transmitting the detailed metadata information in the source library and the detailed metadata information in the target library to the structure difference analysis module; Based on the data differential comparison in the structural difference analysis module, a first structural difference map under the active detection layer is generated; Based on the passive trigger layer and through the SQL exception parser, the data synchronization information between the source database and the target database is processed for exception information capture to obtain structural error information; Transmitting the structural error information to the structural difference analysis module to generate a second structural difference map under the passive trigger layer; The first structural difference map and the second structural difference map are combined to obtain the structural difference map.

3. A method for dynamic synchronization of heterogeneous database structures according to claim 1, characterized in that: According to the initial DDL statement corresponding to the structural difference graph, the abstract syntax tree after syntax analysis in the source database is traversed and converted to generate DDL statements that conform to the target database, and the structural consistency information after the target database executes the structural change is obtained, specifically including: Extracting structural difference information from the structural difference graph through a cross-platform DDL converter, and generating the initial DDL statement to be executed from the structural difference information; Performing syntax analysis on the initial DDL statements in the source library through the AST parsing module to construct the abstract syntax tree; wherein each node in the abstract syntax tree represents a syntax element; According to the abstract syntax tree and the data type rules of the target library, the abstract syntax tree in the source library is traversed and converted to generate DDL statements that conform to the target library; The DDL statement is input into a DDL execution module, and a structure change operation is performed on the target library to make the library structure between the target library and the source library consistent, thereby generating the structure consistency information.

4. A method for dynamic synchronization of heterogeneous database structures according to claim 1, characterized in that: Based on the structural consistency information, incremental synchronization processing is performed on the migration task between the source database and the target database to obtain incremental synchronization information, specifically including: When the structure consistency information is detected, starting the migration task; Performing a full scan on the structure of the source database and transmitting the scanned complete structure information to the target database to initialize the structure of the target database and determine the initial synchronization status information; If the initial synchronization status information is detected, the source database is monitored for data changes in the incremental state to determine incremental change data; wherein the data changes include at least: data insertion, data update, and data parameters; The incremental change data in the source database is synchronized to the target database according to the preset data change rules, and the incremental synchronization information is generated.

5. A method for dynamic synchronization of heterogeneous database structures according to claim 1, characterized in that: If structural difference information is detected in the incremental synchronization information, a structural repair process is performed on the target library to obtain structural repair result information, which specifically includes: If, during the generation of the incremental synchronization information, the multimodal structure synchronization engine detects that there is structure difference information between the source database and the target database, a cross-platform DDL converter generates a repair DDL statement corresponding to the structure difference information, and converts the migration task in the incremental synchronization state into a migration task in the structure repair state; Backing up the current data in the target database; Based on the repair DDL statement, the structure change operation is re-executed on the target library to generate the structure repair result information; wherein, the structure repair result information includes: repair operation failure information and repair operation success information; the repair operation success information corresponds to continuing to execute incremental synchronization; the repair operation failure information corresponds to executing data fuse.

6. A method for dynamic synchronization of heterogeneous database structures according to claim 1, characterized in that: If the structure repair result information indicates that the repair operation failed, data fusing is performed on the migration task, and consistency verification is performed between the source database and the target database in the fusing state until incremental synchronization is re-entered. Specifically, the following steps are performed: If the structure repair result information indicates that the repair operation has failed, suspending data synchronization between the source database and the target database; Recording synchronization error information when the migration task is in a fuse state; wherein the synchronization error information includes at least: error type, error occurrence time, error table, and error field; Through manual intervention, a comprehensive data structure comparison is performed between the source database in the blown state and the target database to obtain consistency verification result information; If the consistency check result information is inconsistent information, continue to perform structure repair processing on the target library until the consistency check result information is consistent information; If the consistency check result information is consistent information, then incremental synchronization processing is continued for the migration task between the source database and the target database, and incremental synchronization information after consistency check is generated.

7. A method for dynamic synchronization of heterogeneous database structures according to claim 1, characterized in that: Based on the structural consistency information, incremental synchronization processing is performed on the migration task between the source database and the target database to obtain incremental synchronization information, further comprising: Recording cursor triplet information of the migration task in incremental synchronization; wherein the cursor triplet information includes: current transaction ID, last success timestamp, and synchronized data fingerprint; The cursor triplet information is synchronously stored in a storage module in real time, so as to implement data reading processing of secondary synchronization of the migration task in the migration interruption state.

8. A method for dynamic synchronization of heterogeneous database structures according to claim 1, characterized in that: If the incremental synchronization information is in a completed state, before marking the migration task as migration completion information, the method further includes: If it is identified that the migration task is in a migration interruption state, the fault interruption type is identified; wherein the fault interruption type includes: network failure and hardware failure; If it is the network failure, then gradually increase the reconnection interval time through the exponential backoff algorithm to determine the waiting reconnection recovery strategy; If it is the hardware failure, then based on the disaster recovery switching rules, switch to the backup environment to continue executing the migration task, and verify the validity of the downstream triplet information when the source database is in the breakpoint state; If the cursor triplet information is valid, the system automatically selects a fault breakpoint recovery strategy based on the interruption duration, the number of changed rows in the source database, and predefined data recovery rules. The fault breakpoint recovery strategies include incremental resume mode, differential patch synchronization mode, and full verification + incremental retransmission mode. If the cursor triplet information is invalid, suspending data synchronization of the migration task; After the waiting-for-reconnection recovery strategy and / or the fault breakpoint recovery strategy execute corresponding operations, a bidirectional verification of the hash values ​​between the source database and the target database is performed to obtain bidirectional verification result information; If the bidirectional verification result information is bidirectionally consistent information, the migration task continues to perform incremental synchronization processing; If the bidirectional verification result information is bidirectional inconsistency information, the migration task is re-performed with the recovery strategy repair process in the fault interruption state until the migration task continues to perform incremental synchronization processing.

9. A dynamic synchronization device for heterogeneous database structures, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the method for dynamic synchronization of heterogeneous database structures according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that The storage medium is a non-volatile computer-readable storage medium, which stores at least one program. Each of the programs includes instructions. When the instructions are executed by the terminal, the terminal executes the method for dynamic synchronization of heterogeneous database structures according to any one of claims 1 to 8.

Images