Database real-time synchronization method, device and equipment based on double-layer breakpoint continuation and dynamic table structure mapping

Abstract

This application provides a method, apparatus, and electronic device for real-time database synchronization based on a two-layer breakpoint resumption mechanism and dynamic table structure mapping. The method extracts persistent real-time incremental synchronization breakpoints and full completion detection breakpoints from a breakpoint storage table, and extracts the change log retention period characteristics of the source database for synchronization state recovery. It captures change events in the source database in real time and encapsulates them into event wrapper objects containing target table mapping information to generate grouped sets of events to be synchronized for different target tables. Sample documents are extracted and dynamic table structure mapping processing is performed to update and persist the latest real-time incremental synchronization breakpoints. This application, through a two-layer breakpoint mechanism and an adaptive mapping engine, solves the technical defects of traditional solutions, such as the difficulty of resuming interrupted data and the technical shortcomings of synchronization interruptions caused by schema changes, effectively ensuring data consistency and synchronization continuity between heterogeneous databases.

Description

Technical Field

[0001] This application relates to the field of database technology, and more specifically, to a method, apparatus, and device for real-time database synchronization based on dual-layer breakpoint resume and dynamic table structure mapping. Background Technology

[0002] In current big data processing architectures, data synchronization between heterogeneous databases is fundamental to achieving data aggregation and real-time analysis. In particular, when performing incremental synchronization from NoSQL databases with flexible schema features (such as MongoDB) to more structured relational databases (such as MySQL), ensuring rapid recovery after abnormal interruptions and maintaining stability of the link when the source table structure evolves are bottleneck problems that urgently need to be solved in the industry.

[0003] In existing heterogeneous database synchronization solutions, an incremental extraction mode based on log listening is typically adopted. This solution first predefines the static table structure mapping relationship of the target database in the synchronization engine; then, it records the position of the currently read change log through a single position logger; finally, it writes the captured incremental data into the target database according to a fixed format.

[0004] However, this traditional synchronization solution has significant technical flaws. The core flaws are: difficulty in recovering from breakpoints and synchronization interruptions due to schema changes. Because existing technologies mostly use single-level log entries, once the service restarts and the log entries expire, the system often needs to be completely rewritten, resulting in low data recovery efficiency. Simultaneously, due to the lack of dynamic awareness of source-side field evolution, when new fields are added or data types are modified on the source side, the synchronization link often crashes and breaks down directly due to incompatibility with the target table structure, failing to meet the continuous synchronization requirements of complex production environments. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a database real-time synchronization method, apparatus, and device based on dual-layer breakpoint resume and dynamic table structure mapping, thereby at least alleviating the aforementioned technical problems.

[0006] A real-time database synchronization method based on dual-level breakpoint resume and dynamic table structure mapping includes: Step 1: Extract persistent real-time incremental synchronization breakpoints and full completion detection breakpoints from the breakpoint storage table, and extract the change log retention period characteristics of the source database to restore the synchronization state. Step 2: In response to the restored synchronization state, capture change events of the source database in real time, and encapsulate each change event into an event wrapper object containing target table mapping information to generate a grouped set of events to be synchronized for different target tables; Step 3: Extract sample documents from the grouped event set to be synchronized and perform dynamic table structure mapping processing to update and persist the latest real-time incremental synchronization breakpoint.

[0007] Optionally, it also includes: step 4, determining the primary key set of the source database based on the full completion detection breakpoint, and persisting the full completion detection breakpoint to the latest primary key position of the currently processed shard in conjunction with the existing primary key set in the target database.

[0008] Optionally, step 1 specifically includes: Based on the service instance identifier, persistent real-time incremental synchronization breakpoints and full completion detection breakpoints are extracted from the breakpoint storage table, and the change log retention period characteristics of the source database are extracted. Based on the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time, the synchronization state is restored based on service instance isolation.

[0009] Optionally, based on the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time, synchronization state recovery based on service instance isolation is performed, including: The difference between the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time is calculated to determine whether the real-time incremental synchronization breakpoint exceeds the change log retention period feature. If the time limit is not exceeded, a change monitoring stream with breakpoint resume function is constructed, and monitoring is resumed from the breakpoint position of the real-time incremental synchronization. Otherwise, monitoring is restarted from the current system time point to perform synchronization state recovery based on service instance isolation.

[0010] Optionally, step 2 specifically includes: In response to the restored synchronization state, change events of the source database are captured in real time through the change listening stream, and each change event is encapsulated into an event wrapper object containing target table mapping information and sent to a bounded buffer queue for backpressure buffering to generate a set of grouped events to be synchronized for different target tables.

[0011] Optionally, generating grouped sets of events to be synchronized for different target tables specifically involves: The timed batch processing component is invoked to extract event payloads from the bounded buffer queue in batches according to a preset time interval, and table-by-table grouping and aggregation are performed on the event payloads to generate a set of grouped events to be synchronized for different target tables.

[0012] Optionally, step 3 specifically includes: Sample documents are extracted from the grouped event set to be synchronized, and dynamic table structure mapping is performed to generate sample document field mappings. A type inference algorithm is used to identify the runtime type of the fields in the sample document field mappings that represent business attribute characteristics, and intelligent field name escaping is performed synchronously to avoid keyword conflicts, so as to generate corresponding target table definition instructions. Based on this, the structure of the target database is dynamically adjusted to implement table structure adaptation. After table structure adaptation, batch update and insert statements are constructed for the grouped event set to be synchronized and loaded into the target database for batch incremental synchronization. After successful synchronization, the latest real-time incremental synchronization breakpoint is updated and persisted.

[0013] Optionally, step 4 specifically includes: The scheduled completion task is invoked, starting from the last primary key position recorded at the full completion detection breakpoint. The primary key set of the source database is queried in segments, and the existing primary key set in the target database is retrieved in batches. A hash lookup structure is used to perform a difference operation on the primary key sets of the source and target databases to identify the list of missing primary keys. Based on the list of missing primary keys, the full document is retrieved from the source database and written in batches to the target database to complete the data completion. Simultaneously, the full completion detection breakpoint is updated and persisted to the latest primary key position of the currently processed segment.

[0014] A real-time database synchronization device based on dual-level breakpoint resume and dynamic table structure mapping includes: The first program unit is used to extract persistent real-time incremental synchronization breakpoints and full completion detection breakpoints from the breakpoint storage table, and to extract the change log retention period characteristics of the source database for synchronization state recovery. The second program unit is used to capture change events of the source database in real time in response to the restored synchronization state, and encapsulate each change event into an event wrapper object containing target table mapping information to generate a grouped set of events to be synchronized for different target tables. The third program unit is used to extract sample documents from the grouped event set to be synchronized and perform dynamic table structure mapping processing to update and persist the latest real-time incremental synchronization breakpoint.

[0015] An electronic device includes a memory and a processor. The memory stores a computer-executable program, and the processor runs the computer-executable program to implement the above-mentioned database real-time synchronization method based on dual-level breakpoint resume and dynamic table structure mapping.

[0016] Technical advantages of the technical solution provided in this application This application provides a real-time database synchronization method based on dual-layer breakpoint resumption and dynamic table structure mapping. Addressing the technical bottlenecks mentioned in the background technology, such as difficulties in breakpoint recovery and synchronization interruptions caused by schema changes, this method achieves efficient management of synchronization points through a dual-layer breakpoint mechanism introduced in step 1: "real-time incremental synchronization breakpoint" and "full completion detection breakpoint." Compared to traditional single-layer breakpoint solutions, this application, by determining the difference between the cluster timestamp and the log retention period, can quickly determine the validity of the point when the service restarts. Combined with the asynchronous completion mechanism in step 4, it solves the data inconsistency problem caused by difficulties in breakpoint recovery in the background technology, ensuring smooth continuation of the synchronization state in abnormal scenarios.

[0017] Meanwhile, this application utilizes runtime type inference and intelligent keyword escaping to generate target table adjustment instructions through dynamic table structure mapping in step 3. This mechanism addresses the technical deficiency of schema change interruption synchronization in the background technology. Compared to traditional solutions that rely on predefined static structures, this application can adapt to the data evolution at the source end in real time. By adaptively adapting the table structure, it ensures the continuous availability of heterogeneous synchronization links. Combined with bounded buffering and batch aggregation in step 2, it greatly improves the synchronization throughput and system robustness under large-scale data flow. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a real-time database synchronization method based on dual-layer breakpoint resume and dynamic table structure mapping, as described in an embodiment of this application.

[0019] Figure 2 This is a structural diagram of a database real-time synchronization device based on dual-layer breakpoint resume and dynamic table structure mapping, according to an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of an electronic device structure according to an embodiment of this application. Detailed Implementation

[0021] like Figure 1 As shown, this application provides a database real-time synchronization method based on dual-layer breakpoint resume and dynamic table structure mapping, including: Step 1: Extract persistent real-time incremental synchronization breakpoints and full completion detection breakpoints from the breakpoint storage table, and extract the change log retention period characteristics of the source database to restore the synchronization state. Step 2: In response to the restored synchronization state, capture change events of the source database in real time, and encapsulate each change event into an event wrapper object containing target table mapping information to generate a grouped set of events to be synchronized for different target tables; Step 3: Extract sample documents from the grouped event set to be synchronized and perform dynamic table structure mapping processing to update and persist the latest real-time incremental synchronization breakpoint.

[0022] Optionally, step 1 specifically includes: Based on the service instance identifier, persistent real-time incremental synchronization breakpoints and full completion detection breakpoints are extracted from the breakpoint storage table, and the change log retention period characteristics of the source database are extracted. Based on the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time, the synchronization state is restored based on service instance isolation.

[0023] Preferably, in the specific implementation of step 1 in this application, the service instance identifier is first read and standardized. The service instance identifier is used to distinguish different synchronous execution instances in a distributed deployment scenario. It adopts a naming convention of environment identifier plus sequence number. The environment identifier is used to distinguish different operating environments such as production, testing, pre-release, and development, and the sequence number is used to distinguish different parallel instances in the same environment. The reading of the service instance identifier follows the priority order of startup parameters, environment variables, and configuration files. After reading, the service instance identifier is format-validated to ensure that it conforms to the naming convention. The service instance identifier that passes the validation is used as the core filtering basis for subsequent breakpoint query, breakpoint writing, and state isolation, thereby preventing the breakpoint records of different synchronous instances from overwriting each other from the root.

[0024] Preferably, in the specific implementation of step 1, based on the verified service instance identifier, the corresponding real-time incremental synchronization breakpoint and full completion detection breakpoint are extracted from the breakpoint storage table. The breakpoint storage table is a pre-built structured storage unit for persistently storing synchronization point information. It contains fields such as source database set name, target database table name, breakpoint type, real-time incremental synchronization breakpoint serialization content, full completion detection breakpoint content, service instance identifier, and update timestamp. It also sets a joint unique constraint on the three fields of source database set name, breakpoint type, and service instance identifier. When performing the breakpoint extraction operation, the service instance identifier is used as the core filtering condition. Two independent query operations are performed for the two breakpoint types, incremental synchronization and full completion, respectively, to obtain the real-time incremental synchronization breakpoint and full completion detection breakpoint corresponding to the current service instance. If the query result has no corresponding record, an empty value is returned as the initial state of the breakpoint of that type.

[0025] Preferably, in the specific implementation of step 1, structured parsing and integrity verification are performed on the extracted real-time incremental synchronization breakpoints. For non-empty real-time incremental synchronization breakpoints obtained from the query, the serialized string content is first parsed to convert it into structured data containing change event location information. Then, integrity verification is performed on the converted structured data. The verification dimensions include whether it contains the source database cluster timestamp field, whether it contains the change event location identifier field, and whether the data structure conforms to the specification requirements of the source database change flow. If a format error occurs during the parsing process, or if the integrity verification results in missing fields or mismatched structures, the real-time incremental synchronization breakpoint is marked as invalid and an empty value is returned as the processing result of the breakpoint, so as to avoid invalid breakpoints causing abnormalities in the subsequent change monitoring process.

[0026] Preferably, in the specific implementation of step 1, the extraction and standardization of the retention period feature of the source database change log are completed. The change log is a fixed-size circular set in the source database replica set used to record all data change operations. Change records that exceed the retention period will be cyclically overwritten by newly generated change records. The change log retention period feature is the longest retention time of change records in this circular set. When performing the extraction operation, a cluster status query command is sent to the management node of the source database to extract the retention period parameter of the change log from the returned cluster status information. If the parameter cannot be obtained from the cluster configuration, the industry-standard default retention period is used as a fallback value. After the extraction is completed, the change log retention period feature is converted into a duration format that is compatible with timestamp comparison, which serves as the benchmark for judging the validity of subsequent real-time incremental synchronization breakpoints.

[0027] Preferably, in the specific implementation of step 1, a breakpoint validity judgment operation is performed based on the cluster timestamp in the real-time incremental synchronization breakpoint and the current system time. From the parsed valid structured real-time incremental synchronization breakpoints, the source database cluster timestamp corresponding to the breakpoint is extracted. This cluster timestamp is used to mark the generation time of the change event corresponding to the breakpoint in the source database change log. The difference between the time corresponding to the cluster timestamp and the current system time is calculated to obtain the time interval between the breakpoint generation time and the current time. This time interval is then compared with the previously extracted change log retention period feature. If the time interval is less than or equal to the change log retention period feature, the real-time incremental synchronization breakpoint is determined to be valid. If the time interval is greater than the change log retention period feature, the real-time incremental synchronization breakpoint is determined to be expired and invalid. At the same time, the corresponding invalid breakpoint record in the breakpoint storage table is cleared.

[0028] Preferably, in the specific implementation of step 1, for the real-time incremental synchronization breakpoint in the valid state, a synchronization state recovery operation based on service instance isolation is performed; when the real-time incremental synchronization breakpoint is determined to be in a valid state, based on the change event position information recorded in the breakpoint, a source database change monitoring stream with breakpoint resume function is constructed, so that the change monitoring stream starts from the change event position marked by the breakpoint and continuously captures the data change events generated by the source database, ensuring that the previous synchronization progress can be resumed after the service restarts, and there will be no duplicate synchronization or data omission; at the same time, the full completion detection breakpoint extracted this time and corresponding to the service instance identifier is loaded into the execution context of the full completion task, providing the starting detection position for subsequent full completion tasks, avoiding the need for the full completion task to scan the full data from the beginning every time.

[0029] Preferably, in the specific implementation of step 1, for real-time incremental synchronization breakpoints in an invalid state, a fallback recovery operation for the synchronization state is performed. When a real-time incremental synchronization breakpoint experiences any invalid situation such as parsing failure, integrity verification failure, or exceeding the change log retention period, the invalid real-time incremental synchronization breakpoint record corresponding to the service instance identifier in the breakpoint storage table is first cleared. Then, a source database change monitoring stream without any breakpoint information is constructed, so that the change monitoring stream starts from the current system time point and captures newly generated data change events in the source database. At the same time, the full completion detection breakpoint corresponding to the service instance identifier is reset to the initial state, triggering a full-scale data completion task to cover data that already exists in the source database during the breakpoint failure period but was not captured by the change monitoring stream, thereby achieving rapid recovery of the synchronization link and fallback support for data consistency.

[0030] Preferably, in the specific implementation of step 1, after the synchronization state recovery is completed, persistent preprocessing of state information based on service instance isolation is performed. After the synchronization state recovery is completed, the running state information of the real-time incremental synchronization breakpoint, full completion detection breakpoint, and change listening stream used in this recovery process is written into the corresponding record in the breakpoint storage table with the service instance identifier as the core associated field, and the latest timestamp is updated for the record. This persistent preprocessing operation provides a basic record carrier for breakpoint updates in the subsequent synchronization process on the one hand, and provides a judgment basis for subsequent instance liveness detection and timeout orphan breakpoint cleanup by updating the timestamp on the other hand. At the same time, relying on the joint unique constraint, it ensures that the state information of different service instances is completely isolated, providing a stable basic support for multi-instance parallel synchronization in distributed deployment scenarios.

[0031] Optionally, based on the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time, synchronization state recovery based on service instance isolation is performed, including: The difference between the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time is calculated to determine whether the real-time incremental synchronization breakpoint exceeds the change log retention period feature. If the time limit is not exceeded, a change monitoring stream with breakpoint resume function is constructed, and monitoring is resumed from the breakpoint position of the real-time incremental synchronization. Otherwise, monitoring is restarted from the current system time point to perform synchronization state recovery based on service instance isolation.

[0032] Preferably, for the parsed real-time incremental synchronization breakpoints corresponding to the current service instance identifier, the cluster timestamp is extracted and standardized. The cluster timestamp is a globally unique time identifier assigned by the source database cluster to each change event, used to mark the event's generation time. It is stored in the structured data of the real-time incremental synchronization breakpoint and is used to locate the physical location of the change event in the source database change log. During the extraction operation, the embedded data segment containing the cluster timestamp is first located in the structured data of the real-time incremental synchronization breakpoint. Then, the second-level time value is extracted from it and converted into a millisecond-level time identifier consistent with the current system time format, resulting in a standardized breakpoint timestamp. This provides a unified format for subsequent time difference calculations. Simultaneously, through the binding relationship between the service instance identifier and the real-time incremental synchronization breakpoint, it ensures that only the breakpoint data corresponding to the current instance is processed, achieving basic instance isolation.

[0033] Preferably, based on the standardized breakpoint timestamp and the current system time, a time difference calculation and preliminary determination of the breakpoint's validity period are performed. First, the current system's millisecond-level time identifier is obtained as the current system time. The standardized breakpoint timestamp is subtracted from the current system time to obtain the time interval between the breakpoint generation time and the current time. Then, the previously extracted source database change log retention period characteristics are converted into a millisecond-level duration threshold with the same format as the time interval value. This duration threshold corresponds to the longest retention period of change records in the source database change log; change records exceeding this duration will be cyclically overwritten by new change records. Subsequently, the calculated time interval value is compared with the duration threshold. If the time interval value is less than or equal to the duration threshold, the real-time incremental synchronization breakpoint is determined to be within its validity period. If the time interval value is greater than the duration threshold, the real-time incremental synchronization breakpoint is determined to have expired. This pre-determination of validity period avoids errors in the listening interface and interruptions in the synchronization process caused by using expired breakpoints.

[0034] Preferably, for real-time incremental synchronization breakpoints within their validity period, a source database change monitoring stream with breakpoint resumption capability is constructed to achieve incremental synchronization state recovery based on service instance isolation. First, based on the current service instance identifier, the mapping relationship between the source database set and target database table responsible for synchronization by this instance is determined. Then, the real-time incremental synchronization breakpoint within its validity period is used as the recovery point parameter of the change monitoring stream and passed to the source database's change stream monitoring interface. Subsequently, the running parameters of the change monitoring stream are configured, including full document return mode, the number of events returned in a single network request, and the timeout duration for long polling. The full document return mode ensures that update operations can return the complete document content after changes, avoiding data inconsistencies caused by missing incremental fields. Then, through the source database's change stream monitoring interface, a change monitoring stream corresponding one-to-one with the current service instance identifier is established. This change monitoring stream starts from the change event position marked by the real-time incremental synchronization breakpoint and continuously captures data change events of the corresponding set in the source database, achieving seamless continuation of incremental synchronization progress after service restart. Simultaneously, by binding the service instance identifier to the change monitoring stream, the monitoring processes of different service instances are completely isolated and do not interfere with each other.

[0035] Preferably, for expired and invalid real-time incremental synchronization breakpoints, breakpoint cleanup and the construction of a pointless change monitoring stream are performed to achieve fallback recovery of the synchronization state. First, using the current service instance identifier as a filter, the corresponding expired real-time incremental synchronization breakpoint record is located in the breakpoint storage table, and the record is cleared to prevent invalid breakpoint information from being read again during subsequent service restarts. Then, a source database change monitoring stream without any recovery point parameters is constructed, configured with runtime parameters consistent with valid breakpoint scenarios. Through the source database change stream monitoring interface, a change monitoring stream bound to the current service instance identifier is established. This change monitoring stream captures newly generated data change events in the corresponding set of the source database starting from the current system time, preventing synchronization process interruptions caused by the use of expired breakpoints. At the same time, a breakpoint failure trigger signal is sent to the execution context of the full completion task, causing the full completion task to reset the full completion detection breakpoint to its initial state, triggering a full-scale data consistency verification and completion operation, covering historical data that existed in the source database during the breakpoint failure period but could not be captured by the new listening stream, thus achieving rapid recovery of the synchronization link and backup support for data consistency.

[0036] Preferably, after constructing the change listening stream, the validity of the listening link and the isolation and persistence of synchronization status information are performed. First, for the newly established change listening stream, connectivity and event capture validity are verified. A heartbeat command is sent to the change listening stream to confirm that the connection between the change listening stream and the source database is normal and that it can receive event data returned by the source database. If the verification fails, retry and recovery processing for the corresponding exception type is performed. Then, the real-time incremental synchronization breakpoints used in this synchronization status recovery process, the running status of the change listening stream, the service instance identifier, and the status update time are integrated into a synchronization status record. Using the service instance identifier as the core associated field, the record is written to the corresponding record in the breakpoint storage table, completing the persistence of the synchronization status. Simultaneously, through the pre-set joint unique constraint in the breakpoint storage table, it is ensured that the synchronization status records of different service instances are independent and will not overwrite each other, providing a fundamental support for the stable operation of multi-instance parallel synchronization in distributed deployment scenarios.

[0037] Preferably, during the operation of the change monitoring stream, dynamic monitoring of the effectiveness of real-time incremental synchronization breakpoints and anomaly adaptation handling based on service instance isolation are performed. First, as the change monitoring stream continuously captures change events, after each batch of change events is processed, the latest cluster timestamp corresponding to that batch of events is extracted, and the difference is calculated with the current system time. The system dynamically monitors whether the position corresponding to that batch of events is close to the threshold for the change log retention period. If it is close to the threshold, a warning message is generated, indicating a risk of excessive latency in the synchronization link. Subsequently, when anomaly information is captured during the operation of the change monitoring stream, key fields are matched to determine whether the anomaly is caused by an expired, invalid, or damaged real-time incremental synchronization breakpoint. If it is determined to be a breakpoint-related anomaly, the current change monitoring stream is immediately terminated, the corresponding invalid breakpoint records are cleared, and a new change monitoring stream starting from the current system time is reconstructed. Simultaneously, a full data completion operation is triggered to ensure data consistency. The entire monitoring and anomaly handling process is based on the current service instance identifier as the boundary. It only processes the change listening stream and breakpoint records corresponding to the current instance and will not affect the synchronization process of other service instances, thus ensuring the independent operation and fault isolation of each synchronization instance in a distributed deployment scenario.

[0038] Preferably, resource isolation and conflict avoidance handling for multi-instance parallel synchronization of change listening streams are performed based on service instance identifiers. First, independent listening thread resources and memory buffer resources are allocated to each service instance, allowing change listening streams corresponding to different service instances to run in independent threads and use independent memory spaces to store captured change events, avoiding resource contention and thread interference between different instances. Then, for deployment scenarios where the source database is a sharded cluster, corresponding sharding filtering rules are configured for each change listening stream based on the service instance identifier. This ensures that each service instance's change listening stream only captures change events within the shard range it is responsible for, reducing cross-shard network transmission and invalid event handling, while avoiding duplicate synchronization issues caused by multiple instances repeatedly capturing the same change event. Furthermore, by binding service instance identifiers to shard ranges, load balancing is achieved among multiple instances, while ensuring that each instance's breakpoint records correspond only to its own responsible shard range. This further strengthens synchronization state management based on service instance isolation, improving the overall throughput and operational stability of the synchronization system in distributed deployment scenarios.

[0039] Optionally, step 2 specifically includes: In response to the restored synchronization state, change events of the source database are captured in real time through the change listening stream, and each change event is encapsulated into an event wrapper object containing target table mapping information and sent to a bounded buffer queue for backpressure buffering to generate a set of grouped events to be synchronized for different target tables.

[0040] Preferably, in the specific technical implementation of step 2, based on the completed synchronization state, continuous capture and processing of source database change events is performed through the constructed change listening stream. The change listening stream establishes a long connection with the corresponding set in the source database, pre-configuring the number of events returned in a single network request, the full document return mode, and the long polling timeout duration. The full document return mode ensures that the change event corresponding to the data update operation includes the complete updated document content, not just the differences in updated fields, avoiding data inconsistency issues caused by missing incremental fields. The change listening stream continuously monitors data change operations generated by the source database in a blocking waiting manner. Data change operations include four types: data insertion, data update, data replacement, and data deletion. Each time a data change operation occurs, the source database generates a corresponding change event, which includes an operation type identifier, the complete updated document content, the document primary key identifier, and the event's corresponding location identifier information. After capturing a change event, the change monitoring stream first performs an integrity check on the change event, checking whether the change event contains complete document content and primary key identifier. If the check passes, the change event is sent to the subsequent encapsulation processing stage. If the check fails, the corresponding exception information is recorded and invalid events are discarded, ensuring that change events entering the subsequent process have complete synchronizable attributes.

[0041] Preferably, in the specific technical implementation of step 2, standardized encapsulation processing is performed on the verified change events to generate an event wrapper object containing target table mapping information. First, based on the mapping relationship between the source database set corresponding to the current service instance identifier and the target database table, the target database table name corresponding to the source database set to which the current change event belongs is found from the mapping configuration. Then, the original content of the change event, the source database set name, the target database table name, the event capture time, and the event location identifier information are integrated and encapsulated into a standardized event wrapper object. The event wrapper object adopts a unified data structure, converting heterogeneous change events into a standardized data carrier that can be uniformly identified in subsequent processing flows. Simultaneously, through the built-in target table mapping information, it provides a direct classification basis for subsequent table-by-table grouping and aggregation processing, avoiding repeated mapping relationship queries in subsequent processing and improving processing efficiency.

[0042] Preferably, in the specific technical implementation of step 2, the encapsulated event packaging object is sent to a pre-initialized bounded buffer queue to complete the temporary storage of change events and traffic peak shaping. The bounded buffer queue adopts a blocking queue structure based on the first-in-first-out rule, with a fixed upper limit for the queue capacity set in advance. The upper limit of the queue capacity is configured according to the event generation rate and batch processing capability of the business peak. For example, it can buffer peak event traffic within a few seconds, reserving sufficient response time for batch processing on the consumer side, and buffering a large number of change events generated by instantaneous peaks to avoid memory overflow and event loss caused by the event processing speed not keeping up with the event generation speed. The enqueue operation of the event packaging object adopts a blocking write method. When the number of elements in the bounded buffer queue has not reached the capacity limit, the event packaging object is directly written to the tail of the queue; when the bounded buffer queue is full, the write operation will enter a blocking waiting state until free storage space appears in the queue, and then the write of the event packaging object will be completed. Basic traffic backpressure control is achieved from the write stage to balance the rate of event production and consumption.

[0043] Preferably, in the specific technical implementation of step 2, a full-link backpressure buffering process is performed based on the blocking mechanism of a bounded buffer queue to adapt to the dynamic fluctuations in event production rate and consumption processing capacity. The backpressure buffering process is divided into two stages: write-end backpressure control and consumer-end rate adaptation. Write-end backpressure control is implemented through the blocking write mechanism of the bounded buffer queue. When the consumer processing speed decreases, causing queue accumulation, the write operation of the event capture thread is automatically blocked, pausing the retrieval of new change events from the change listening stream to avoid memory resource exhaustion caused by unlimited event retrieval. Consumer-end rate adaptation is achieved by dynamically adjusting the batch processing interval and the number of events processed at one time. When the amount of events accumulated in the queue exceeds a preset accumulation threshold, the batch extraction interval is shortened and the number of events extracted at one time is increased to improve the consumption processing rate. When the number of events in the queue is lower than a preset low-water mark threshold, the default batch processing parameters are restored to reduce the write pressure on the target database. Through bidirectional rate adjustment, a dynamic balance between event production and consumption is achieved, ensuring the stable operation of the synchronous link.

[0044] Preferably, in the specific technical implementation of step 2, a timed batch processing component is invoked to extract event payloads from a bounded buffer queue in batches according to a preset fixed time interval, providing a data foundation for subsequent grouping and aggregation processing. The timed batch processing component employs a fixed-frequency triggering mechanism. The triggering time interval can be configured according to the real-time requirements and processing performance of the business. Each time it is triggered, the queue's batch transfer method is called, atomically transferring the event wrapper objects in the bounded buffer queue to the local event processing list, up to a preset maximum batch size. This completes the batch extraction of event payloads. The maximum batch size can be configured according to the write performance of the target database, balancing the database pressure of a single write with the throughput of synchronous processing. The batch transfer operation is atomic, ensuring that event wrapper objects are not extracted repeatedly or lost during the extraction process. Simultaneously, the non-blocking extraction method does not affect the event capturing thread's writing of new event wrapper objects to the bounded buffer queue, achieving parallel execution of event production and consumption.

[0045] Preferably, in the specific implementation of step 2, the batch-extracted event payloads are subjected to table-by-table grouping and aggregation processing to generate grouped event sets to be synchronized for different target tables. First, based on the target database table name contained within the event wrapper object, all batch-extracted event wrapper objects are traversed, and event wrapper objects with the same target database table name are grouped into the same group, forming multiple event groups based on target tables. Then, deduplication and aggregation processing is performed on the event wrapper objects within each event group. Based on the document primary key identifier within the event wrapper object, for multiple event wrapper objects corresponding to the same primary key, only the event wrapper object with the latest capture time is retained, and historical change events with the same primary key are removed, avoiding repeated processing of the same document and ensuring the idempotency of data writing. Finally, the multiple event groups that have undergone deduplication and aggregation processing are integrated into grouped event sets to be synchronized for different target tables, and sent to the subsequent table structure mapping and data writing processing stages, providing standardized data input organized by table for subsequent batch data processing.

[0046] Preferably, in the specific technical implementation of step 2, throughout the entire process of event capture, encapsulation, enqueueing, and group aggregation, exception handling based on service instance isolation is performed to ensure the continuous and stable operation of the synchronization link. In the change event capture stage, if network interruption, connection timeout, or abnormal interruption of the change listening stream is captured, the corresponding exception information is recorded first. Then, based on the real-time incremental synchronization breakpoint corresponding to the current service instance identifier, the construction and recovery operation of the change listening stream is re-executed. After recovery, event capture continues, without affecting the processing of events already written to the bounded buffer queue. In the event enqueueing and batch extraction stage, if queue operation exceptions or incorrect event packaging object formats occur, the abnormal event packaging objects are isolated and recorded, without affecting the normal event processing flow. In the group aggregation stage, if the target table mapping relationship is missing, the corresponding exception information is recorded and the event processing of that group is paused, waiting for the mapping relationship to be supplemented before resuming subsequent operations. This does not affect the normal processing of other target table corresponding groups. Through segmented exception isolation, single-point exceptions are prevented from causing the interruption of the entire synchronization link.

[0047] Optionally, generating grouped sets of events to be synchronized for different target tables specifically involves: The timed batch processing component is invoked to extract event payloads from the bounded buffer queue in batches according to a preset time interval, and table-by-table grouping and aggregation are performed on the event payloads to generate a set of grouped events to be synchronized for different target tables.

[0048] Preferably, in the specific implementation process of generating grouped sets of events to be synchronized for different target tables, the initialization and runtime parameter configuration of the timed batch processing component are completed first, providing standardized execution rules for subsequent event batch extraction and group aggregation. The timed batch processing component adopts a fixed-frequency timed triggering mechanism. The triggering interval can be configured according to the business's requirements for real-time synchronization and the write performance of the target database. At the same time, the component is configured with a maximum number of events to be processed in a single batch. This number is set according to the batch write capacity of the target database to limit the upper limit of the event payload extracted from the bounded buffer queue during a single trigger, avoiding excessive write pressure on the target database due to excessively large data volume processed in a single batch. The operation of the component is bound to the current service instance identifier and only processes the event payload in the bounded buffer queue corresponding to the current service instance, ensuring that the batch processing process of different service instances is completely isolated in multi-instance distributed deployment scenarios, and there will be no problem of duplicate event extraction and cross-processing.

[0049] Preferably, after the timed batch processing component completes triggering at preset time intervals, it performs a batch extraction operation of event payloads from the bounded buffer queue to obtain standardized event data to be processed. The batch transfer method of the bounded buffer queue is called, with the maximum number of events to be processed in a single batch configured during initialization as the upper limit, atomically transferring the event wrapper objects in the bounded buffer queue to the component's local event processing list, thus completing the batch extraction of event payloads. The batch transfer method is atomic, ensuring that event wrapper objects are not extracted repeatedly or lost during the transfer process. Simultaneously, the non-blocking extraction method does not block the event capture thread from writing new event wrapper objects to the bounded buffer queue, achieving parallel execution of event production and consumption. If the number of event wrapper objects in the bounded buffer queue is less than the maximum number of events to be processed in a single batch, all event wrapper objects in the queue are transferred to the event processing list; if the bounded buffer queue is empty, the subsequent grouping and aggregation operations are not performed during this trigger, and the system directly waits for the next trigger, avoiding unnecessary idle runs that consume computing resources.

[0050] Preferably, after batch extraction of event payloads, pre-processing integrity checks and anomaly filtering are performed on the event wrapper objects in the event processing list to remove invalid event payloads and ensure the data validity of subsequent grouping and aggregation processing. The checks include whether the event wrapper object contains complete target table mapping information, a valid document primary key identifier, complete change event content, and whether it belongs to the source database set that the current service instance is responsible for synchronizing. Event wrapper objects that pass the integrity check are retained in the event processing list for subsequent grouping and aggregation. Invalid event wrapper objects that fail the check are written to the anomaly log along with their corresponding anomaly information, event capture time, and set information, and are simultaneously removed from the event processing list to prevent invalid events from causing anomalies in subsequent grouping, aggregation, and data writing processes. For data deletion type change events, a separate event type identifier is used to ensure that they are differentiated from data insertion, update, and replacement type events in subsequent grouping and aggregation processes, providing a basis for subsequent differentiated write operations.

[0051] Preferably, after completing the pre-validation and filtering of the event payload, the target table mapping information within the event wrapping object is used as the core basis to perform table-by-table grouping and aggregation processing on the event wrapping objects in the event processing list, forming multiple event groups based on the target database table. First, all valid event wrapping objects in the event processing list are traversed, and the target database table name recorded in each event wrapping object is extracted. Using this table name as the grouping key, event wrapping objects with the same target database table name are grouped into the same group. Each group corresponds to a unique target database table, and the group contains all the event wrapping objects to be processed corresponding to that table. This table-by-table grouping and aggregation process organizes the originally unordered event payload into an ordered manner according to the target table, allowing subsequent table structure adaptation and data writing operations to be executed independently by table. Event groups corresponding to different target tables do not interfere with each other, and a synchronization failure in one table will not affect the synchronization process of other tables. Simultaneously, it provides standardized data input organized by table for subsequent batch writing operations, improving the execution efficiency of batch writing.

[0052] Preferably, after completing table-by-table grouping and aggregation, for the event wrapper objects within the event group corresponding to each target table, deduplication based on the same primary key and merging of operation types are performed to eliminate duplicate change operations on the same document and ensure idempotency of data writing. Based on the document primary key identifier within the event wrapper object, event wrapper objects within the same event group are aggregated. For multiple event wrapper objects corresponding to the same document primary key, they are sorted in ascending order according to the event capture timestamp, retaining only the event wrapper object with the latest capture time in the sorting result, and removing all historical change events corresponding to that primary key. This avoids repeated processing of the same document and reduces invalid database write operations. For events with the same primary key, if the last event is a data deletion type, all preceding change events corresponding to that primary key are directly removed, retaining only the event wrapper object of the data deletion type; if the last event is a data insertion, update, or replacement type, the complete document content within that event is used as the final data to be synchronized, without the need to overlay preceding change operations, ensuring that the data written to the target database is the latest state of the document and avoiding data inconsistency issues caused by multiple change operations.

[0053] Preferably, after completing the deduplication of primary keys and the merging of operation types, operation type classification and structure standardization are performed on the event groups corresponding to each target table to generate standardized event groups adapted to subsequent table structure mapping and batch write processes. The event wrapper objects within each event group are divided into two categories according to operation type: the first category is data insertion, update, and replacement events, and the second category is data deletion events. For the first category of events, the complete document content, document primary key identifier, and event position information within each event wrapper object are extracted and integrated into a list of documents to be written, providing sample documents and data to be synchronized for subsequent dynamic table structure mapping processing. For the second category of events, the document primary key identifier and event position information within each event wrapper object are extracted and integrated into a list of primary keys to be deleted, providing a basis for subsequent target database data deletion operations. Simultaneously, each event group is appended with the corresponding target database table name, the name of its source database set, and the current service instance identifier information to ensure that the ownership and processing boundaries of each event group are clear and that cross-table and cross-instance processing confusion does not occur.

[0054] Preferably, after standardizing all event groups, multiple standardized event groups based on target tables are integrated into a set of events to be synchronized for different target tables, completing the entire process of batch event extraction and group aggregation. Each sub-item in the set of events to be synchronized corresponds to a unique target database table, containing the list of documents to be written, the list of primary keys to be deleted, and the event position range information. This provides complete, table-organized standardized data input for subsequent dynamic table structure mapping, batch data writing, and breakpoint update operations. When the set of events to be synchronized is sent to the subsequent processing flow, the total number of events processed in this batch, the number of target tables involved, and the time range of event capture are recorded synchronously for monitoring the running status and performance statistics of the synchronization link. At the same time, the maximum event position information corresponding to this batch processing is bound to the execution result of subsequent data writing. Only after all events in the set of events to be synchronized in this batch have been processed will the real-time incremental synchronization breakpoint be updated based on the maximum event position information, ensuring the consistency between breakpoint updates and data writing and avoiding data loss or duplicate synchronization issues.

[0055] Optionally, step 3 specifically includes: Sample documents are extracted from the grouped event set to be synchronized, and dynamic table structure mapping is performed to generate sample document field mappings. A type inference algorithm is used to identify the runtime type of the fields in the sample document field mappings that represent business attribute characteristics, and intelligent field name escaping is performed synchronously to avoid keyword conflicts, so as to generate corresponding target table definition instructions. Based on this, the structure of the target database is dynamically adjusted to implement table structure adaptation. After table structure adaptation, batch update and insert statements are constructed for the grouped event set to be synchronized and loaded into the target database for batch incremental synchronization. After successful synchronization, the latest real-time incremental synchronization breakpoint is updated and persisted.

[0056] Preferably, in the specific technical implementation of step 3, firstly, for each target table corresponding to the event group within the grouped event set to be synchronized, sample document extraction and standardized parsing processing are performed to generate sample document field mappings. For the list of documents to be written within each event group, the document with the most complete field coverage is selected as the sample document. If it is the first synchronization of the target table, the first document within the event group is selected as the sample document. All field names and corresponding field values ​​are extracted from the sample document, excluding the primary key field used to uniquely identify the document, and a one-to-one mapping relationship between field names and field values ​​is constructed to generate sample document field mappings. At the same time, the table structure cache manager is invoked to query the existing structure cache information of the corresponding table in the target database. If the cache information does not exist or has expired, the latest table structure information is queried from the system table of the target database and updated to the cache, providing a benchmark for subsequent field matching and structure adjustment. The generated sample document field mappings will serve as the core input for subsequent type inference and field name escaping processing.

[0057] Preferably, for the generated sample document field mapping, the runtime type of each field used to characterize business attribute features in the sample document field mapping is identified by the designed type inference algorithm, so as to determine the target database data type corresponding to each field. The type inference algorithm follows a fixed priority rule. First, it performs whitelist matching of the date and time field. If the field name belongs to the predefined whitelist, the target database type of the field is directly determined to be date and time, regardless of the actual runtime type of the field value. Second, it performs null value handling. If the field value is null, the target database type of the field is determined to be a variable-length character type with character set settings to ensure maximum compatibility. Next, it performs runtime type matching. Based on the runtime type of the field value, it matches the predefined type mapping rules to convert different runtime types to the corresponding target database data types. For example, integer types are mapped to integers, long integer types are mapped to long integers, boolean types are mapped to single-byte integers, and complex types such as nested documents, arrays, and key-value pairs are mapped to long text types. Finally, it performs string length judgment. If the field value is a string type, the corresponding type is determined based on the actual length of the string. If the length exceeds a preset threshold, it is mapped to a long text type; otherwise, it is mapped to a variable-length character type with a reserved length buffer. The reserved length buffer is used to avoid storage truncation problems caused by subsequent data length growth.

[0058] Preferably, while executing the type inference algorithm, intelligent escaping processing is performed on each original field name in the sample document field mapping to avoid syntax errors caused by conflicts with reserved keywords in the target database and illegal characters. First, illegal character replacement is performed on the original field names, uniformly replacing all non-alphanumeric, non-underscore characters with underscores. Second, a standardized prefix is ​​added to field names starting with numbers to avoid syntax errors caused by field names starting with numbers. Then, empty field names are replaced with preset placeholders, and excessively long field names are truncated and have a hash suffix added to ensure that the field name length meets the column name length limit of the target database while guaranteeing the uniqueness of the field names. Next, the processed field names are converted to uppercase and matched against a pre-maintained set of reserved keywords in the target database to check if the field name is a reserved keyword. Finally, all processed field names are uniformly enclosed in backticks to completely avoid syntax errors caused by reserved keywords. After processing, a one-to-one correspondence between the original field names and the escaped column names is established and updated in the sample document field mapping. This mapping relationship will provide a basis for field matching during subsequent data writing.

[0059] Preferably, based on the sample document field mapping that has completed type inference and field name escaping, and combined with the existing structure information of the corresponding table in the target database, a corresponding target table definition instruction is generated, providing an executable operational basis for structural adjustments to the target database. First, based on the existing structure information of the target table, it is determined whether the target table exists. If the target table does not exist, a table creation instruction is generated, fixing the type and primary key constraint of the primary key field. Then, based on the content of the sample document field mapping, the column name and corresponding data type definition of each field are sequentially concatenated, while simultaneously setting the table's storage engine and character set configuration. If the target table already exists, the fields in the sample document field mapping are compared with the existing column information of the target table to identify newly added fields that do not exist in the target table, generating a column addition instruction. Simultaneously, the actual type of the existing column is compared with the inferred field type, and compatibility is determined according to a preset type compatibility matrix. If incompatible, a column type modification instruction is generated according to the rule of promoting to the most compatible type. For example, when a numeric type conflicts with a string type, it is promoted to a variable-length character type; when a date type conflicts with a string type, it is promoted to a short-length variable-length character type. All generated target table definition instructions are executed in a safe mode if they do not exist, to avoid syntax errors caused by repeated execution.

[0060] Preferably, the generated target table definition command is sent to the target database for execution to dynamically adjust the table structure of the target database and implement table structure adaptation processing to match the field evolution of the source database. Before executing the target table definition command, the table structure cache corresponding to the target table is cleared to avoid errors in subsequent processing caused by old structure information in the cache. When executing the table creation command, if the target table already exists, it is skipped; otherwise, the table is created according to the command. When executing the column addition command, if the corresponding column already exists, it is skipped; otherwise, the column is added according to the command. When executing the column type modification command, the column type is modified according to the type promotion rules to ensure that the modified type is compatible with historical data and new data. After all commands are executed, the latest structure information of the table is retrieved from the system table of the target database, updated to the table structure cache, and the cache expiration time is set. At the same time, the audit log of the table structure change is recorded. Table structure adaptation is performed independently for each target table. An abnormal adjustment of the structure of a single target table will not affect the synchronous processing of other target tables. After the table structure adaptation is completed, the target table can fully adapt to the field characteristics of the source document, thus removing the obstacle of structural incompatibility for subsequent batch data writing.

[0061] Preferably, after adapting the target table structure, for each target table's corresponding event group within the set of events to be synchronized, batch update and insert statements are constructed and loaded into the target database for execution to complete the batch incremental synchronization of the corresponding data. For the list of documents to be written within the event group, combining the mapping relationship between the original field names and escaped column names, the field values ​​of each document are mapped one-to-one with the columns of the target table, constructing a standardized list of field values. Batch update and insert statements are constructed based on this list, employing an execution semantic of updating if the data exists and inserting if it doesn't, using the document primary key as the conflict judgment criterion to ensure idempotency of data writing. For data deletion events within the event group, batch delete statements are constructed based on the document primary key. When executing batch statements, batch statement rewriting optimization of the target database is enabled, merging multiple single-line write statements into a single multi-value write statement, reducing network round trips and improving data writing throughput. If a batch statement execution fails, it automatically degrades to a line-by-line execution mode, skipping conflicting exception records to ensure that most normal data can be successfully written, while recording exception information for execution failures to provide a basis for subsequent troubleshooting.

[0062] Preferably, after all event groups corresponding to all target tables within the grouped event set to be synchronized have completed batch incremental synchronization, the latest event position information corresponding to this synchronization process is extracted, and the latest real-time incremental synchronization breakpoint is updated and persisted to the breakpoint storage table. First, from all event wrapper objects processed in this phase, the position information corresponding to the event with the latest capture time is extracted as the latest real-time incremental synchronization breakpoint. The real-time incremental synchronization breakpoint is serialized into a storable text format. Using the current service instance identifier, source database set name, and incremental synchronization breakpoint type as filtering conditions, update or insert operations are performed on the breakpoint storage table. A data processing-before-breakpoint persistence execution strategy is adopted to ensure that breakpoint information is only updated after all data in the corresponding batch has been successfully written to the target database, avoiding data loss due to breakpoint update completion but data write failure. The pre-defined joint unique constraint in the breakpoint storage table ensures that breakpoint records from different service instances, different source database sets, and different types are isolated from each other, preventing breakpoint records from overwriting each other. After the real-time incremental synchronization breakpoints are persisted, the number of events processed in this synchronization, the number of target tables involved, the processing time, and other runtime monitoring information are recorded to complete the entire process of step 3. At the same time, it provides an effective breakpoint basis for the restoration of the synchronization state when the service is restarted.

[0063] Optionally, it also includes: step 4, determining the primary key set of the source database based on the full completion detection breakpoint, and persisting the full completion detection breakpoint to the latest primary key position of the currently processed shard in conjunction with the existing primary key set in the target database.

[0064] Preferably, in the specific implementation of step 4, the trigger initialization of the scheduled completion task and the reading and parsing of the breakpoint for full completion detection are completed first, providing a starting point for subsequent primary key comparison and data completion. The scheduled completion task adopts a fixed-period timed triggering mechanism, and also supports a manual triggering execution mode. The execution cycle of the task can be configured according to the data consistency requirements of the business and the database operating load, prioritizing execution during off-peak business periods to reduce the performance impact on the source and target databases. After the task is triggered, the service instance identifier corresponding to the current synchronization task is first obtained. Using the service instance identifier, the name of the corresponding collection in the source database, and the type of full completion breakpoint as filtering conditions, the corresponding full completion detection breakpoint is queried from the breakpoint storage table. If the query result is empty, the full completion detection breakpoint is set to an initial null value, which means that the detection starts from the beginning position of the corresponding collection in the source database. If a non-empty full completion detection breakpoint is found, the breakpoint content is parsed, and the last primary key value of the breakpoint record is converted into a data type that matches the primary key format of the source database. At the same time, the validity of the primary key value is verified, providing accurate starting filtering conditions for subsequent sharded queries.

[0065] Preferably, based on the parsed full completion detection breakpoint, a sharded query of the primary key set of the corresponding set in the source database is performed to obtain the source primary key data to be compared. First, query filtering conditions for the source database are constructed based on the parsing results of the full completion detection breakpoint. If the full completion detection breakpoint is an initial null value, no primary key filtering conditions are set. If the full completion detection breakpoint is a valid primary key value, a filtering condition is set where the primary key value is greater than the breakpoint value, ensuring that only primary key data after the last detection end position is queried, avoiding full duplicate scanning. During the query process, only the primary key field is projected, and other business fields of the document are not returned to reduce network transmission overhead and memory usage. Simultaneously, the query results are sorted in ascending order according to the primary key value to ensure the continuity of the sharded query. Then, the query results are sharded according to a preset shard size, and a single query returns only a fixed number of primary key data, forming the source database primary key list for the current shard. This sharded query mode breaks down the full data comparison into multiple small batches of shards, which can effectively reduce the performance impact of a single query on the source database, while avoiding memory overflow issues in massive data scenarios.

[0066] Preferably, for each shard of the source database, the primary key list obtained from the query is processed by batch retrieval and standardization of the existing primary key set in the corresponding table of the target database, providing benchmark data for subsequent difference operations. First, the primary key list of the source database for the current shard is standardized and converted to string format to eliminate comparison errors caused by differences in primary key data types between the source and target databases. Then, the standardized primary key list is split according to a preset maximum batch size, forming multiple batch query units to avoid syntax errors and performance degradation caused by excessively long single query statements. For each batch query unit, a primary key query statement for the target database is constructed to retrieve existing primary key values ​​from the corresponding table. The primary key results returned by all batch queries are merged and stored in a primary key set constructed based on a hash lookup structure. This hash lookup structure reduces the time complexity of primary key existence checks to constant level, significantly improving comparison efficiency in scenarios with massive primary keys compared to traditional list traversal comparison methods.

[0067] Preferably, based on the source database shard primary key list and the existing primary key hash set in the target database, a difference operation is performed to identify a list of missing primary keys that exist in the source database but are missing in the target database. First, the source database primary key list of the current shard is traversed. For each standardized primary key value, the existence check method of the hash lookup structure is called to determine if the primary key value exists in the existing primary key hash set in the target database. If the check result is no, the primary key value is added to the missing primary key list; if the check result is yes, the primary key value is skipped. After traversal, the obtained missing primary key list is validated to remove null values ​​and abnormally formatted primary key data, ultimately forming the valid missing primary key list for the current shard. This difference operation can accurately locate missing data on the target database caused by abnormal scenarios such as network fluctuations, service restarts, and expired breakpoints, providing a precise target range for subsequent data completion. It eliminates the need for field-by-field comparison of all documents, significantly reducing the computational overhead of data consistency checks.

[0068] Preferably, for the identified list of missing primary keys, a full document lookup in the source database and batch write processing to the target database are performed to complete the missing data completion operation. First, the list of missing primary keys is converted to a format matching the primary key type in the source database. Batch query conditions are constructed to retrieve the complete business documents corresponding to these primary keys from the corresponding set in the source database, obtaining a list of documents to be completed. Then, dynamic table structure mapping processing logic is invoked to perform field type inference and intelligent field name escaping for each document to be completed, ensuring that the document's field structure matches the target database table structure and avoiding write failures due to field incompatibility. Subsequently, batch update and insert statements are constructed based on the adapted document list, employing an execution semantic of updating if the document exists and inserting if it does not, ensuring idempotency of data writing. The statements are loaded into the target database for batch write execution. If batch write execution fails, it automatically degrades to a line-by-line write mode, skipping individual documents with anomalies, recording exception information while ensuring successful writing of most normal documents. Finally, the number of data entries successfully completed for the current shard is counted and recorded.

[0069] Preferably, after completing the missing data completion operation for the current shard, the full completion detection breakpoint is updated and persisted, persisting the full completion detection breakpoint to the latest primary key position of the currently processed shard. First, the last primary key value in the primary key list of the source database for the current shard is extracted as the updated full completion detection breakpoint value. Then, using the service instance identifier, the corresponding collection name of the source database, and the full completion breakpoint type as filtering conditions, update or insert statements are constructed for the breakpoint storage table, persisting the latest full completion detection breakpoint value to the breakpoint storage table. The pre-defined composite unique index in the breakpoint storage table ensures that breakpoint records from different service instances, different collections, and different types are isolated from each other, preventing breakpoint overwriting. Simultaneously, an execution strategy of completing shard data processing first and then updating breakpoints is adopted. Breakpoint persistence is only performed after all data completion operations for the current shard are completed, ensuring that execution can continue from the latest breakpoint position after an abnormal interruption, without repeatedly processing already verified shard data, effectively improving the execution efficiency of the completion task.

[0070] Preferably, after completing the sharded loop processing of the full completion detection breakpoint, task termination and breakpoint status management operations are performed to ensure the integrity of the full completion process and the executability of subsequent tasks. After completing the breakpoint update of the current shard, it is determined whether the primary key list of the source database for the current shard is empty. If it is empty, it means that there is no more primary key data to be detected in the corresponding set of the source database. The current full completion task is completed, and the corresponding full completion detection breakpoint record in the breakpoint storage table is cleared to avoid repeating historical data processing when the next task starts. If the primary key list of the current shard is not empty, the query, comparison, completion, and breakpoint update process for the next shard continues until all shards are processed. Throughout the sharded loop processing, the update of the full completion detection breakpoint and the update of the real-time incremental synchronization breakpoint are completely independent. The two types of breakpoints serve the two scenarios of data consistency completion and real-time incremental synchronization respectively, without interfering with each other, forming a two-layer independent breakpoint management system. This effectively solves the technical defect of traditional single-layer breakpoint solutions that cannot simultaneously handle incremental synchronization and full completion breakpoint recovery.

[0071] Preferably, during the entire process of full completion detection breakpoint handling, exception tolerance and resource isolation processing based on service instance identifiers are implemented to ensure the stable operation of the completion task in a distributed deployment scenario. First, an exception capture and retry mechanism is set up at each stage of task execution. For transient exceptions such as network fluctuations, database connection interruptions, and query timeouts, an exponential backoff retry strategy is adopted. The current operation is re-executed after waiting for a fixed period of time to avoid the overall interruption of the task due to transient exceptions. If the retry still fails after multiple attempts, the exception information is recorded and the processing of the current shard is paused, waiting for the next task cycle to continue execution, without affecting the completion tasks of other sets or other shards. Meanwhile, in a distributed multi-instance deployment scenario, each service instance only reads and updates the full completion detection breakpoint corresponding to its own service instance identifier. The completion tasks of different instances are completely isolated, and there will be no issues such as breakpoint records being overwritten or tasks being executed repeatedly. In addition, a resource isolation mechanism is set up between the completion task and the real-time incremental synchronization task to limit the upper limit of the amount of data queried and written by the completion task in a single instance, so as to avoid the completion task consuming too much database resources and network bandwidth, and to ensure that the running performance and synchronization latency of the real-time incremental synchronization task are not affected.

[0072] Optionally, step 4 specifically includes: The scheduled completion task is invoked, starting from the last primary key position recorded at the full completion detection breakpoint. The primary key set of the source database is queried in segments, and the existing primary key set in the target database is retrieved in batches. A hash lookup structure is used to perform a difference operation on the primary key sets of the source and target databases to identify the list of missing primary keys. Based on the list of missing primary keys, the full document is retrieved from the source database and written in batches to the target database to complete the data completion. Simultaneously, the full completion detection breakpoint is updated and persisted to the latest primary key position of the currently processed segment.

[0073] Preferably, in the specific technical implementation of step 4, the triggering initialization of the scheduled completion task and the reading and parsing of the breakpoints for full completion detection are completed first, providing an accurate starting point for subsequent primary key consistency comparison and missing data completion. The scheduled completion task adopts a fixed-period timed triggering mechanism, and also supports a manually triggered execution mode. The execution cycle of the task can be configured according to the data consistency requirements of the business and the database operating load, prioritizing execution during off-peak business periods to reduce the performance impact on the source and target databases. After the task is triggered, the service instance identifier corresponding to the current synchronization task is first obtained. Using the service instance identifier, the corresponding collection name in the source database, and the full completion breakpoint type as filtering conditions, the corresponding full completion detection breakpoint is queried from the breakpoint storage table. If the query result is empty, the full completion detection breakpoint is set to an initial null value, which means that the consistency check starts from the beginning position of the corresponding collection in the source database. If a non-null full completion detection breakpoint is found, the breakpoint content is parsed, and the last primary key value of the breakpoint record is converted into a data type that matches the primary key format of the source database. At the same time, the legality and validity of the primary key value are verified, and invalid breakpoint content such as abnormal format and null value is removed, providing accurate starting filtering conditions for subsequent sharded queries.

[0074] Preferably, based on the parsed full completion detection breakpoint, a sharded query process is performed on the primary key set of the corresponding set in the source database to obtain the standardized source primary key data to be compared. First, query filtering conditions for the source database are constructed based on the parsing results of the full completion detection breakpoint. If the full completion detection breakpoint is an initial null value, no primary key filtering conditions are set. If the full completion detection breakpoint is a valid primary key value, a filtering condition is set where the primary key value is greater than the breakpoint value, ensuring that only primary key data after the last detection end position is queried, avoiding database performance loss caused by repeated full scans. During the query process, only the primary key field is projected, and other business fields of the document are not returned to reduce network transmission overhead and memory resource consumption. Simultaneously, the query results are sorted in ascending order according to the primary key value to ensure the continuity and traceability of the sharded query. Then, the query results are sharded according to a preset shard size, and a single query returns only a fixed number of primary key data, forming the source database primary key list for the current shard. This sharded query mode breaks down the full data consistency comparison into multiple small-batch sharded processing units, which can effectively reduce the performance impact of a single query on the source database, while avoiding memory overflow issues in massive data scenarios and improving the operational stability of the completion task.

[0075] Preferably, for each shard of the source database, the primary key list obtained from the query is processed by batch retrieval and standardization of the existing primary key set in the corresponding table of the target database, providing reliable comparison benchmark data for subsequent difference operations. First, the primary key list of the source database for the current shard is standardized and converted to string format to eliminate comparison errors caused by differences in primary key data types between the source and target databases. Then, the standardized primary key list is split according to a preset maximum batch size, forming multiple batch query units to avoid syntax errors and performance degradation caused by excessively long single query statements. For each batch query unit, a primary key query statement for the target database is constructed to retrieve existing primary key values ​​from the corresponding table. The primary key results returned by all batch queries are merged and stored in the target database primary key set constructed based on a hash lookup structure. This hash lookup structure reduces the time complexity of primary key existence checks to constant level. Compared to the traditional list traversal comparison mode, it significantly improves the comparison efficiency in scenarios with massive primary keys and reduces the computational resource overhead of consistency checks.

[0076] Preferably, based on the source database shard primary key list and the existing primary key hash set in the target database, a difference operation is performed to accurately identify the list of missing primary keys that exist in the source database but are missing in the target database. First, the source database primary key list of the current shard is traversed. For each standardized primary key value, the existence check method of the hash lookup structure is called to determine whether the primary key value exists in the existing primary key hash set in the target database. If the check result is no, the primary key value is added to a temporary missing primary key list; if the check result is yes, the primary key value is skipped without further processing. After traversal, the resulting temporary missing primary key list is validated to remove invalid primary key data such as null values, abnormal formats, and primary key length limits, ultimately forming the valid missing primary key list for the current shard. This difference operation can accurately locate missing data on the target database caused by network fluctuations, service restarts, breakpoint expiration, instantaneous write anomalies, etc., providing a precise target range for subsequent data completion. It eliminates the need for field-by-field comparison of all documents, significantly reducing the computational overhead and time cost of data consistency verification.

[0077] Preferably, for the identified list of missing primary keys, a full document lookup in the source database and batch write processing to the target database are performed to complete the missing data completion operation. First, the list of missing primary keys is converted into a format that matches the primary key type of the source database. Batch query conditions are constructed to query the complete business documents corresponding to these primary keys from the corresponding set in the source database, obtaining a list of documents to be completed. Then, the dynamic table structure mapping processing logic is called to perform field runtime type inference and intelligent field name escaping for each document to be completed, ensuring that the field structure of the document is fully compatible with the table structure of the target database, avoiding write failures caused by field incompatibility, keyword conflicts, or type mismatches. Subsequently, based on the adapted document list, batch update and insert statements are constructed, employing the execution semantics of updating if the document exists and inserting if it does not. The document primary key is used as the basis for conflict judgment to ensure the idempotency of data writing and avoid data inconsistency caused by duplicate writes. The statements are then loaded into the target database for batch writing. If the batch write execution fails, it automatically degrades to the individual write mode, skipping individual documents with anomalies, recording the corresponding error information, while ensuring the successful writing of most normal documents. Finally, the number of data entries successfully completed in the current shard and the error data information are counted and recorded.

[0078] Preferably, after completing the missing data completion operation for the current shard, the full completion detection breakpoint is updated and persisted, persisting the full completion detection breakpoint to the latest primary key position of the currently processed shard. First, the last primary key value in the primary key list of the current shard's source database is extracted as the updated full completion detection breakpoint value. Then, using the service instance identifier, the corresponding source database set name, and the full completion breakpoint type as filtering conditions, update or insert statements are constructed for the breakpoint storage table, persisting the latest full completion detection breakpoint value to the breakpoint storage table. The pre-defined joint unique constraint in the breakpoint storage table ensures that breakpoint records from different service instances, different source database sets, and different types are isolated from each other, preventing breakpoint records from overwriting each other. Simultaneously, the execution strategy of completing shard data processing first and then updating breakpoints is adopted. Breakpoint persistence is only executed after all data completion operations for the current shard are completed, ensuring that execution can continue from the latest breakpoint position after an abnormal interruption, without repeatedly processing already verified shard data, effectively improving the execution efficiency and fault tolerance of the completion task.

[0079] Preferably, after completing the full completion and breakpoint update for the current shard, perform shard loop verification and full task cleanup management operations to ensure the integrity of the full completion process and the executability of subsequent tasks. After completing the breakpoint update for the current shard, check if the primary key list of the source database for the current shard is empty. If it is empty, it means that there is no more primary key data to be detected in the corresponding set of the source database. The current full completion task is completed, and the corresponding full completion and breakpoint record in the breakpoint storage table is cleared to avoid reprocessing historical data when the next task starts. If the primary key list of the current shard is not empty, continue to execute the primary key query, consistency comparison, data completion, and breakpoint update process for the next shard until all shards are processed. Throughout the entire sharding loop processing, the updates of the full completion detection breakpoint and the real-time incremental synchronization breakpoint are completely independent. The two types of breakpoints serve the two business scenarios of data consistency completion and real-time incremental synchronization respectively, without interfering with each other, forming a two-layer independent breakpoint management system. This effectively solves the technical defect of the traditional single-layer breakpoint solution that cannot simultaneously take into account the recovery of incremental synchronization breakpoints and the resumption of full completion breakpoints.

[0080] Preferably, during the entire process of full completion detection breakpoint handling, exception tolerance and resource isolation processing based on service instance identifiers are implemented to ensure the stable operation of the completion task in a distributed deployment scenario. First, an exception capture and retry mechanism is set up at each stage of task execution. For transient exceptions such as network fluctuations, database connection interruptions, and query timeouts, an exponential backoff retry strategy is adopted. The current operation is re-executed after waiting for a fixed period of time to avoid the overall interruption of the task due to transient exceptions. If the retry still fails after multiple attempts, detailed exception information is recorded and the processing of the current shard is paused, waiting for the next task cycle to continue execution, without affecting the completion tasks of other source database sets or other shards. Meanwhile, in a distributed multi-instance deployment scenario, each service instance only reads and updates the full completion detection breakpoint corresponding to its own service instance identifier. The completion tasks of different instances are completely isolated, and there will be no issues such as breakpoint records being overwritten or tasks being executed repeatedly. In addition, a resource isolation mechanism is set up between the completion task and the real-time incremental synchronization task to limit the upper limit of the amount of data queried and written by the completion task in a single instance, so as to avoid the completion task consuming too much database resources and network bandwidth, and to ensure that the running performance and synchronization latency of the real-time incremental synchronization task are not affected.

[0081] Preferably, the shard update and persistence of the full completion detection breakpoint adopts an execution strategy of "completing shard data processing first, then updating breakpoints." The breakpoint update and persistence process is only triggered after all data completion operations for the current shard have been successfully completed. This ensures consistency between the breakpoint record and the actual data processing progress, avoiding data loss or duplicate processing issues caused by premature breakpoint updates. After all operations such as missing primary key list reverse lookup, document structure adaptation, and batch data writing for the current shard are completed, the number of successfully completed data entries and exception information for the current shard are counted. Once it is confirmed that there are no fatal exceptions, the breakpoint update process is started. If there are unresolved write exceptions in the current shard, the breakpoint update is paused, the original breakpoint position is retained, and the process continues only after the exception is resolved.

[0082] Preferably, when an update operation triggers a full completion check breakpoint, the last primary key value in the primary key list of the current shard source database is first extracted as the updated full completion check breakpoint value. This primary key value is the maximum value among all primary keys in the current shard, sorted in ascending order. It marks the furthest position the consistency check has reached in the corresponding set of the source database. When the next completion task starts, the query will continue from the next position after this primary key value, avoiding repeated scanning of shard data that has already been verified. After extracting the primary key value, it is formatted and converted into a format compatible with the field types in the breakpoint storage table, ensuring the smooth execution of subsequent persistence operations.

[0083] Preferably, based on the updated full completion detection breakpoint value, an update or insert statement is constructed for the breakpoint storage table, and the breakpoint persistence operation is executed. Using the service instance identifier, the corresponding collection name in the source database, and the full completion breakpoint type as joint filtering conditions, the system first checks if a corresponding breakpoint record already exists in the breakpoint storage table. If it exists, an update statement is constructed to update the last primary key value in the breakpoint record to the latest primary key value of the current shard, and simultaneously update the last modification time of the breakpoint. If it does not exist, an insert statement is constructed to integrate the service instance identifier, the corresponding collection name in the source database, the full completion breakpoint type, the current latest primary key value, the creation time, and the modification time into a new breakpoint record, which is then inserted into the breakpoint storage table. This update or insert operation is executed atomically to ensure the uniqueness and consistency of the breakpoint records.

[0084] Preferably, the pre-defined joint unique constraint in the breakpoint storage table provides data isolation guarantees for the sharded updates and persistence of breakpoints detected during full completion. The joint unique constraint consists of three fields: service instance identifier, source database set name, and full completion breakpoint type. This ensures that breakpoint records from different service instances, different source database sets, and different types are independent of each other in the storage table, preventing breakpoint records from overwriting each other. In a distributed multi-instance deployment scenario, each service instance can only read and update the breakpoint record corresponding to its own service instance identifier. Even if multiple instances simultaneously process completion tasks for the same source database set, task chaos will not occur due to breakpoint conflicts, effectively ensuring the stable operation of completion tasks in a distributed environment.

[0085] Preferably, the shard update and persistence operations of the full completion detection breakpoint are closely coordinated with the shard loop processing flow to form a continuous consistency detection chain. After completing the breakpoint persistence of the current shard, it is immediately determined whether the primary key list of the source database of the current shard is empty. If it is empty, it means that there is no more primary key data to be detected in the corresponding set of the source database, and the current full completion task is completed. At this time, the corresponding full completion detection breakpoint record in the breakpoint storage table is cleared to avoid repeating the processing of historical data when the next task starts. If the primary key list of the current shard is not empty, it means that there is still undetected data in the source database. The primary key query, consistency comparison, data completion and breakpoint update process of the next shard is immediately started until all shards are processed. The entire chain is closely linked to ensure the integrity of the full data consistency detection.

[0086] Preferably, during the full completion and detection breakpoint shard update and persistence process, an exception capture and retry mechanism is set up to ensure the reliability of the breakpoint record. For update or insert operations on the breakpoint storage table, if execution fails due to momentary exceptions such as network fluctuations, database connection interruptions, or lock conflicts, an exponential backoff retry strategy is adopted. The process first waits for a short fixed duration before re-executing the breakpoint persistence operation. If it fails again, the waiting time is extended, and the attempts are repeated multiple times until the operation succeeds or the maximum number of retries is reached. If the maximum number of retries is reached and the operation still fails, detailed exception information is recorded, the current completion task is paused, and the original breakpoint record is retained unchanged. This ensures that after the task restarts, execution can continue from the original breakpoint position, and data processing progress will not be lost due to breakpoint update failure.

[0087] Preferably, the combined unique constraint of the breakpoint storage table consists of three fields: service instance identifier, corresponding set name of the source database, and full completion breakpoint type. This combination of three fields serves as the key value of the combined unique index, uniquely identifying each full completion detection breakpoint record at the database storage level. When inserting or updating a breakpoint record, the database system first performs a uniqueness check on the key value combination of the record to be written. If a record with the same key value combination already exists in the table, the insertion of the new record is rejected, or only the existing record corresponding to that key value combination is allowed to be updated. If no record with the same key value combination exists in the table, the insertion of the new record is allowed. This enforces the uniqueness of the key value combination from the underlying storage mechanism, fundamentally preventing accidental overwriting of breakpoint records.

[0088] Preferably, in a distributed multi-instance deployment scenario, the joint unique constraint uses the service instance identifier field to achieve breakpoint isolation between different service instances. Each service instance has a globally unique service instance identifier, and the identifier values ​​of different service instances are completely different. Even if multiple service instances simultaneously perform full completion tasks on the same set in the source database, the combined key-value pairs formed will be completely different due to their different service instance identifiers. The database system will identify the breakpoints of different service instances as independent records, store and update them separately. The breakpoint records written by the service instance that starts later will not overwrite the existing records of the service instance that started earlier, ensuring that the completion tasks of each service instance in the distributed environment are completely isolated and do not interfere with each other.

[0089] Preferably, the unique constraint uses a full completion breakpoint type field to isolate different types of breakpoints. This field distinguishes breakpoint records for different business purposes, such as full completion detection breakpoints and real-time incremental synchronization breakpoints. Even within the same service instance and the same source database set, different breakpoint types will result in different combined key-value pairs. The database system identifies different types of breakpoints as independent records and stores them separately, preventing accidental overwriting between full completion detection breakpoints and real-time incremental synchronization breakpoints. This ensures the independence of the two types of breakpoints in the two-layer breakpoint mechanism, guaranteeing that breakpoint management for the real-time incremental synchronization and full completion business processes does not interfere with each other.

[0090] Preferably, for the same service instance, the same source database corresponding set, and the same full completion breakpoint type, the joint unique constraint allows updates to existing records, ensuring the normal progress of the breakpoint. In this case, the key-value combination is unique. When the application layer performs an update operation, the database system will only update the existing records corresponding to that key-value combination, updating the last primary key value to the latest position of the current shard, and simultaneously updating the last modification time, without affecting records with other key-value combinations. This design ensures that breakpoints for the same completion task within the same instance can be updated normally with the shard processing progress, while preventing chaotic overwriting between breakpoints of different instances, different sets, and different types, achieving precise management of "the same task can proceed, different tasks are isolated."

[0091] Preferably, the execution of the joint unique constraint and the application-layer field validation form a dual protection, further strengthening the anti-overwriting effect. When constructing the breakpoint record to be written, the application layer strictly follows the field requirements of the joint unique constraint to ensure that the values ​​of the three fields—service instance identifier, corresponding collection name of the source database, and breakpoint type—are accurate and formatted correctly, avoiding key-value combination chaos caused by incorrect field values. The joint unique constraint of the database system, as the underlying mandatory guarantee, works in conjunction with the application-layer pre-validation to jointly prevent accidental overwriting of breakpoint records from both the business logic and storage logic levels, providing reliable support for the stable operation of the dual-layer breakpoint resume mechanism.

[0092] like Figure 2 As shown, this application provides a database real-time synchronization device based on dual-layer breakpoint resume and dynamic table structure mapping, which includes: The first program unit is used to extract persistent real-time incremental synchronization breakpoints and full completion detection breakpoints from the breakpoint storage table, and to extract the change log retention period characteristics of the source database for synchronization state recovery. The second program unit is used to capture change events of the source database in real time in response to the restored synchronization state, and encapsulate each change event into an event wrapper object containing target table mapping information to generate a grouped set of events to be synchronized for different target tables. The third program unit is used to extract sample documents from the grouped event set to be synchronized and perform dynamic table structure mapping processing to update and persist the latest real-time incremental synchronization breakpoint.

[0093] The device also includes a fourth program unit, which is used to determine the primary key set of the source database based on the full completion detection breakpoint, and persist the full completion detection breakpoint to the latest primary key position of the currently processed shard in conjunction with the existing primary key set in the target database.

[0094] like Figure 3As shown, this application provides an electronic device including a processor and a memory; the memory is used to store a computer program; the processor is used to implement the steps of any of the methods described in this application when running the computer program.

[0095] It should be noted that the type of database in the above embodiments is not limited to a single type; it can be, but is not limited to, MongoDB / MySQL, as long as it can achieve the ideas of this application.

Claims

1. A real-time database synchronization method based on dual-layer breakpoint resume and dynamic table structure mapping, characterized in that, include: Step 1: Extract persistent real-time incremental synchronization breakpoints and full completion detection breakpoints from the breakpoint storage table, and extract the change log retention period characteristics of the source database to restore the synchronization state. Step 2: In response to the restored synchronization state, capture change events of the source database in real time, and encapsulate each change event into an event wrapper object containing target table mapping information to generate a grouped set of events to be synchronized for different target tables; Step 3: Extract sample documents from the grouped event set to be synchronized and perform dynamic table structure mapping processing to update and persist the latest real-time incremental synchronization breakpoint.

2. The method for real-time synchronization of heterogeneous databases based on dual-layer breakpoint resume and dynamic table structure mapping according to claim 1, characterized in that, It also includes: Step 4, determining the primary key set of the source database based on the full completion detection breakpoint, and persisting the full completion detection breakpoint to the latest primary key position of the currently processed shard in conjunction with the existing primary key set in the target database.

3. The method for real-time synchronization of heterogeneous databases based on dual-layer breakpoint resume and dynamic table structure mapping according to claim 1, characterized in that, Step 1 is as follows: Based on the service instance identifier, persistent real-time incremental synchronization breakpoints and full completion detection breakpoints are extracted from the breakpoint storage table, and the change log retention period characteristics of the source database are extracted. Based on the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time, the synchronization state is restored based on service instance isolation.

4. The method for real-time synchronization of heterogeneous databases based on dual-layer breakpoint resume and dynamic table structure mapping according to claim 3, characterized in that, Based on the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time, perform synchronization state recovery based on service instance isolation, including: The difference between the cluster timestamp recorded in the real-time incremental synchronization breakpoint and the current system time is calculated to determine whether the real-time incremental synchronization breakpoint exceeds the change log retention period feature. If the time limit is not exceeded, a change monitoring stream with breakpoint resume function is constructed, and monitoring is resumed from the breakpoint position of the real-time incremental synchronization. Otherwise, monitoring is restarted from the current system time point to perform synchronization state recovery based on service instance isolation.

5. The method for real-time synchronization of heterogeneous databases based on dual-layer breakpoint resume and dynamic table structure mapping according to claim 1, characterized in that, Step 2 is as follows: In response to the restored synchronization state, change events of the source database are captured in real time through the change listening stream, and each change event is encapsulated into an event wrapper object containing target table mapping information and sent to a bounded buffer queue for backpressure buffering to generate a set of grouped events to be synchronized for different target tables.

6. The method for real-time synchronization of heterogeneous databases based on dual-layer breakpoint resume and dynamic table structure mapping according to claim 5, characterized in that, The specific steps for generating grouped sets of events to be synchronized for different target tables are as follows: The timed batch processing component is invoked to extract event payloads from the bounded buffer queue in batches according to a preset time interval, and table-by-table grouping and aggregation are performed on the event payloads to generate a set of grouped events to be synchronized for different target tables.

7. The method for real-time synchronization of heterogeneous databases based on dual-layer breakpoint resume and dynamic table structure mapping according to claim 1, characterized in that, Step 3 specifically involves: Sample documents are extracted from the grouped event set to be synchronized, and dynamic table structure mapping is performed to generate sample document field mappings. A type inference algorithm is used to identify the runtime type of the fields in the sample document field mappings that represent business attribute characteristics, and intelligent field name escaping is performed synchronously to avoid keyword conflicts, so as to generate corresponding target table definition instructions. Based on this, the structure of the target database is dynamically adjusted to implement table structure adaptation. After table structure adaptation, batch update and insert statements are constructed for the grouped event set to be synchronized and loaded into the target database for batch incremental synchronization. After successful synchronization, the latest real-time incremental synchronization breakpoint is updated and persisted.

8. The method for real-time synchronization of heterogeneous databases based on dual-layer breakpoint resume and dynamic table structure mapping according to claim 1, characterized in that, Step 4 is as follows: The scheduled completion task is invoked, starting from the last primary key position recorded at the full completion detection breakpoint. The primary key set of the source database is queried in segments, and the existing primary key set in the target database is retrieved in batches. A hash lookup structure is used to perform a difference operation on the primary key sets of the source and target databases to identify the list of missing primary keys. Based on the list of missing primary keys, the full document is retrieved from the source database and written in batches to the target database to complete the data completion. Simultaneously, the full completion detection breakpoint is updated and persisted to the latest primary key position of the currently processed segment.

9. A database real-time synchronization device based on dual-layer breakpoint resume and dynamic table structure mapping, characterized in that, include: The first program unit is used to extract persistent real-time incremental synchronization breakpoints and full completion detection breakpoints from the breakpoint storage table, and to extract the change log retention period characteristics of the source database for synchronization state recovery. The second program unit is used to capture change events of the source database in real time in response to the restored synchronization state, and encapsulate each change event into an event wrapper object containing target table mapping information to generate a grouped set of events to be synchronized for different target tables. The third program unit is used to extract sample documents from the grouped event set to be synchronized and perform dynamic table structure mapping processing to update and persist the latest real-time incremental synchronization breakpoint.

10. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer-executable program, and the processor is used to run the computer-executable program to implement the database real-time synchronization method based on dual-layer breakpoint resume and dynamic table structure mapping as described in any one of claims 1-8.

Images