High-robustness data real-time synchronization method and system, storage medium and electronic equipment

Abstract

The invention provides a high-robustness data real-time synchronization method and system, a storage medium and electronic equipment, relates to the technical field of data synchronization, and can significantly reduce the data synchronization delay between different systems by introducing mixed timestamps and fragment increment verification and calculation and adopting a reliable transmission mechanism and idempotent processing protocols based on message queues, thereby improving the data synchronization performance. Almost real-time information sharing is realized; meanwhile, the data consistency of the cross-heterogeneous information system can be effectively improved, and the accuracy and integrity of the data among multi-source systems are ensured; in addition, the reliability and robustness of data synchronization in unstable environments such as limited network bandwidth, easy jitter or interruption and the like can be greatly enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and more specifically, to a highly robust real-time data synchronization method, system, storage medium, and electronic device. Background Art

[0002] Currently, existing technologies face severe challenges in achieving efficient, accurate, and reliable synchronization of key medical data, such as health records, across different institutions and heterogeneous systems. These challenges mainly include the following: 1) Data synchronization delay: Data synchronization between county-level medical institutions often relies on traditional batch processing or scheduled polling mechanisms, such as nightly batch synchronization or polling for changes every few hours. This results in the target system not receiving the latest data until minutes or even hours after the source data is updated. This delay seriously affects the timeliness of clinical decision-making (for example, the lack of timely access to a patient's latest medical history during an emergency referral) and also hinders the development of health management and public health monitoring applications based on real-time data.

[0003] 2) Data consistency challenges across heterogeneous systems: Medical information systems within county-level medical communities are often developed by different software vendors, utilizing diverse databases, data models, coding standards, and interface specifications. Traditional data extraction, transformation, and loading tools (such as ETL tools) or point-to-point interfaces often encounter errors, omissions, duplications, and inconsistencies during data synchronization due to complex data conversion logic, inconsistent standards, and inadequate exception handling. This leads to low consistency rates for synchronized data. This directly impacts data credibility and availability, hindering cross-institutional collaboration and the effectiveness of regional data analysis.

[0004] 3) Insufficient data synchronization reliability in unstable network environments: Network infrastructure in county-level areas is relatively weak, often facing issues such as limited bandwidth, network jitter, and transmission interruptions. Traditional, simple data transmission methods based on TCP / IP are prone to packet loss, connection timeouts, and synchronization task failures in unstable network environments. Furthermore, the lack of effective breakpoint resumption, error retry, and data verification mechanisms makes it difficult to ensure the success rate and integrity of data synchronization, which in turn affects the continuity of medical services.

[0005] Common data synchronization solutions, such as incremental synchronization based on database triggers and scheduled tasks, change data capture (CDC) based on database logs, traditional ETL tools, and periodic full data comparison, still have the following shortcomings when addressing the above issues: 1) Scheduled full / incremental synchronization: This method cannot meet near-real-time synchronization requirements and results in high latency. Full comparison consumes a lot of resources.

[0006] 2) Traditional ETL tools: Complex configuration, high cost of adapting to heterogeneous systems, poor real-time performance, and reliance on complex transformation rules to ensure consistency.

[0007] 3) Basic CDC solutions based on database logs: Although they can capture changes, they often lack consideration for data transmission, consistency verification, and robustness assurance in unreliable network environments. They are also inefficient in processing complex data types (such as large field data).

[0008] Therefore, there is an urgent need for a new data synchronization technology solution that can effectively reduce data synchronization delays between county medical institutions, improve data consistency across heterogeneous systems, and enhance synchronization reliability in unstable network environments. Summary of the Invention

[0009] In order to overcome the problems of large data synchronization delay, difficulty in ensuring data consistency between heterogeneous systems, and poor synchronization reliability when the network is unstable in the above-mentioned existing technologies, the present invention provides a highly robust real-time data synchronization method, system, storage medium and electronic equipment, aiming to build an efficient, reliable and consistent data exchange and sharing foundation for county-level medical communities, and support the coordinated development of various businesses such as clinical diagnosis and treatment, public health, and health management.

[0010] In order to solve the above technical problems, the technical solutions of the present invention are as follows: A highly robust real-time data synchronization method, which uses message-based middleware to synchronize data between a data source and a data target, includes the following steps: S1: At the data source end, the transaction log of the source database is monitored in real time. In response to the triggering of a data change event, the corresponding change data is captured and pre-processed; S2: Generate a hybrid timestamp for the pre-processed change data, where the hybrid timestamp includes a physical timestamp and a logical sequence number; S3: Perform data sharding on the changed data that meets the preset sharding conditions after preprocessing to obtain a plurality of sharded data; calculate an incremental checksum for each of the sharded data and / or unsharded changed data; S4: Encapsulate the pre-processed change data together with the mixed timestamp, incremental checksum, and pre-assigned globally unique message ID into a message, and send the message to a message queue preset in the message middleware; S5: At the data target end, obtain a message from the message queue and perform an idempotency check based on the message ID in the message. If the idempotency check passes, execute step S6; otherwise, discard the message and obtain a new message. S6: Recalculate the incremental checksum of the message and compare it with the incremental checksum encapsulated in the message. If the comparison result meets the preset conditions, the check passes and step S7 is executed. For messages that have undergone data fragmentation, data reassembly is performed first and then step S7 is executed. Otherwise, the check fails, the message is discarded, and a new message is obtained. S7: Determine the latest version of the message according to the mixed timestamp, execute a corresponding data change event based on the latest version of the message, and complete the data synchronization.

[0011] Preferably, in step S1, the transaction log of the source database is monitored in real time by using CDC technology, and the change data is captured; The preprocessing at least includes: parsing and format conversion of the captured change data.

[0012] Preferably, in step S2, the logical sequence number is a sequence number that increases monotonically within the same physical timestamp; First, the physical timestamp is generated, and then the logical sequence number is generated; if the newly generated physical timestamp is the same as the physical timestamp of the last changed data, the logical sequence number is incremented by a preset value; when the physical timestamp is updated, the corresponding logical sequence number is reset to a preset initial value; In step S7, the physical timestamps between the messages are compared first, and the message with the latest physical timestamp is determined as the latest version of the message; if the physical timestamps are the same, the logical sequence numbers between the messages are further compared, and the message with the largest logical sequence number is determined as the latest version of the message.

[0013] Preferably, in step S3, the preset sharding conditions include: the pre-processed change data is large field data of a preset type and / or batch change data; The data sharding process includes: dividing the pre-processed change data into a number of fixed-size shard data; or, hierarchically dividing the pre-processed change data based on the data logical structure to obtain shard data of a number of different levels.

[0014] Preferably, in step S3, any one of xxHash algorithm, CRC32 algorithm, MD5 algorithm, SHA-1 algorithm and Rabin fingerprint algorithm is used to calculate the incremental checksum.

[0015] Preferably, after receiving the message sent by the data source, the message middleware returns an ACK confirmation signal to the data source. If the message fails to be sent or the ACK confirmation signal is not returned due to timeout, the data source resends the message. After executing the corresponding data change event, the data target sends an ACK confirmation signal to the message middleware; The message middleware is provided with a dead letter queue for storing discarded messages and messages that have reached a preset maximum number of sending times.

[0016] Preferably, in step S6, a state library that can be updated in real time is provided in the data target end, and the state library at least stores the message ID of the message corresponding to the executed data change event; If the message ID of the current message already exists in the state library, the idempotency check fails; otherwise, the idempotency check passes.

[0017] The present invention also provides a highly robust real-time data synchronization system, comprising a data source end, a message middleware, and a data target end, wherein the data source end includes the following modules: Change capture module: This module monitors the transaction logs of the source database in real time and, in response to data change events, captures the corresponding change data and performs preprocessing. Message construction and sending module: used to generate a hybrid timestamp for the pre-processed change data; perform data sharding on the pre-processed change data that meets the preset sharding conditions to obtain a plurality of sharded data; calculate an incremental checksum for each sharded data and / or unsharded change data; and encapsulating the pre-processed change data together with the mixed timestamp, incremental checksum, and pre-assigned globally unique message ID into a message, and sending the message to a message queue preset in the message middleware; The data target end includes the following modules: Idempotence check module: used to obtain messages from the message queue and perform idempotence check based on the message ID in the message; Verification and comparison module: used to recalculate the incremental checksum of the message and compare it with the incremental checksum encapsulated in the message; Data synchronization module: used to determine the latest version of the message according to the mixed timestamp, execute the corresponding data change event based on the latest version of the message, and complete data synchronization.

[0018] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps in the above method when executed by a processor.

[0019] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the above method are executed.

[0020] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention provides a highly robust real-time data synchronization method, system, storage medium, and electronic device. By introducing a hybrid timestamp, combining sharded incremental checksums with CDC optimization, and adopting a reliable transmission mechanism based on message queues and an idempotent processing protocol, the present invention can significantly reduce data synchronization latency between different systems and achieve near-real-time information sharing. At the same time, the present invention can effectively improve data consistency across heterogeneous information systems and ensure the accuracy and integrity of data across multi-source systems. In addition, the present invention can significantly enhance the reliability and robustness of data synchronization in unstable environments such as those with limited network bandwidth, prone to jitter or interruptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a highly robust real-time data synchronization method provided in Example 1.

[0022] Figure 2 This is an overall architecture diagram of a highly robust real-time data synchronization method provided in Example 2.

[0023] Figure 3 This is a flow chart of hybrid timestamp generation and management provided in Example 2.

[0024] Figure 4 This is a workflow diagram for the shard incremental verification provided in Example 2.

[0025] Figure 5 This is a flowchart of the reliable transmission and idempotent processing based on the message queue provided in Example 2.

[0026] Figure 6 This is a structural diagram of a highly robust real-time data synchronization system provided in Example 3. DETAILED DESCRIPTION

[0027] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present application; In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size; It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0028] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0029] Example 1 like Figure 1 As shown, this embodiment provides a highly robust real-time data synchronization method, which implements data synchronization between a data source and a data target through a message middleware, including the following steps: S1: At the data source end, the transaction log of the source database is monitored in real time. In response to the triggering of a data change event, the corresponding change data is captured and pre-processed; S2: Generate a hybrid timestamp for the pre-processed change data, where the hybrid timestamp includes a physical timestamp and a logical sequence number; S3: Perform data sharding on the changed data that meets the preset sharding conditions after preprocessing to obtain a plurality of sharded data; calculate an incremental checksum for each of the sharded data and / or unsharded changed data; S4: Encapsulate the pre-processed change data together with the mixed timestamp, incremental checksum, and pre-assigned globally unique message ID into a message, and send the message to a message queue preset in the message middleware; S5: At the data target end, obtain a message from the message queue and perform an idempotency check based on the message ID in the message. If the idempotency check passes, execute step S6; otherwise, discard the message and obtain a new message. S6: Recalculate the incremental checksum of the message and compare it with the incremental checksum encapsulated in the message. If the comparison result meets the preset conditions, the check passes and step S7 is executed. For messages that have undergone data fragmentation, data reassembly is performed first and then step S7 is executed. Otherwise, the check fails, the message is discarded, and a new message is obtained. S7: Determine the latest version of the message according to the mixed timestamp, execute a corresponding data change event based on the latest version of the message, and complete the data synchronization.

[0030] In the specific implementation process, this embodiment is explained using the health record data between county-level medical institutions as an example.

[0031] On the data source side, perform the following steps: First, based on CDC technology, the transaction logs (such as Binlog, Redo Log, and Logical Replication Slot) of the source database (such as HIS or EMR database) are monitored in real time. When a data change event is triggered, the corresponding changed data (including data before and after the change) is captured and preprocessed. In this embodiment, preprocessing can include parsing the captured raw data and converting its format (such as uniformly converting it to JSON format). Then, a globally unique hybrid timestamp is generated for each piece of changed data. In this embodiment, the hybrid timestamp includes a physical timestamp and a logical sequence number. Then, the incremental checksum is calculated. For large data objects or batch-changed data, the data is first sharded, and then the incremental checksum is calculated for each shard. For other regular data, the complete incremental checksum is directly calculated. The message containing the mixed timestamp, sharding information (if any), incremental checksum, actual data (i.e., pre-processed change data), and a globally unique message ID is then encapsulated into a message and sent to the message queue through a reliable protocol.

[0032] In the message middleware, perform the following operations: Message middleware serves as a transit station for data synchronization between the data source and the data target. In this embodiment, a highly reliable, high-throughput message queue system (such as Apache Kafka and Pulsar) is selected, and persistent storage, message confirmation mechanism, partitioning, and replication are configured to ensure that messages are not lost.

[0033] On the data target side, do the following: First, subscribe to and pull messages from the message queue and perform idempotency checks; For data that passes the idempotence check, the integrity of the data is further verified by recalculating the incremental checksum of the message and comparing it with the original checksum in the message. For messages that have been fragmented, they are cached after passing the verification and reassembled in order after all data fragments arrive. After verification is passed and the data is complete, the new and old versions of the data are determined based on the mixed timestamp in the message, and the latest version of the message is determined. At the same time, potential concurrency conflicts are handled based on "latest comer first" or other specific business rules. Finally, the data changes corresponding to the latest version of the message are applied to the data target end to complete data synchronization.

[0034] The innovation of the method provided in this embodiment is: 1) This method innovatively proposes a hybrid timestamp mechanism (high-precision physical timestamp + logical sequence number). This mechanism is more robust and accurate for conflict-prone scenarios commonly encountered in medical data synchronization, such as high-concurrency updates (such as multiple departments simultaneously updating different parts of the same patient medical record), cross-institutional data merging (where server clocks across institutions are difficult to perfectly synchronize), and time-sensitive services (such as the order in which medication orders are executed). For example, when processing resident physical examination data uploaded by multiple primary health care centers within a county-level medical community, due to differences in equipment and network conditions among the centers, the time when the data arrives at the platform may be out of order, and the clocks of the servers may have slight deviations. This mechanism effectively resolves the "false conflicts" and "version overwrites" problems caused by clock inconsistencies in heterogeneous medical systems by combining physical timestamps that reflect real-world time with logical sequence numbers that ensure strict local order, supplemented by clear conflict resolution rules (for example, when the physical timestamp difference is within a threshold, the record with the higher logical sequence is given priority). It is expected to reduce the error rate of data version judgment caused by clock issues from approximately 5% to below 0.01% from the traditional reliance on a single timestamp.

[0035] 2) This method optimizes the transmission of medical big data through the synergistic effect of sharding incremental verification and CDC. In medical data, structured small data (such as vital signs) and unstructured / semi-structured big data often coexist (such as metadata summaries of DICOM images, long free-text medical records, and the text portion of gene sequencing reports). This method innovatively combines the transaction-level change information captured by CDC with dynamic content sharding and efficient incremental verification algorithms. This combined strategy can intelligently identify the type and size of changed data: for small, high-frequency changes (such as blood sugar and blood pressure values), CDC directly captures and quickly verifies them; for large objects (such as partial modifications to a medical record document of several MB), C After DC identifies a change, it does not verify and transmit the entire document. Instead, it combines the sharding mechanism to perform incremental verification, calculation, and transmission of only the shards containing the modified content. For example, in a 10MB medical record document, if only 1KB of content is modified, the traditional method may need to transmit and verify the entire 10MB file, while this method may only need to process several (for example, 64KB) shards containing this 1KB change. This will shorten the difference location and verification time from the average 10 minutes of the traditional method to 10 seconds when synchronizing changes with an average size of several KB to hundreds of KB in large medical record collections of more than 1GB, reducing network bandwidth consumption by 90%, significantly improving synchronization efficiency and reducing system load.

[0036] 3) The message queue-based reliable transmission protocol designed by this method not only relies on the basic guarantees of the message queue itself, but also improves robustness by introducing application-layer enhancement mechanisms. These enhancement mechanisms include: generating a globally unique ID for each business operation (rather than just a message-level ID), carrying refined retry metadata in the message body (such as the current number of retries and the maximum number of retries), maintaining a persistent "processed business ID" state storage on the target end, and dynamically adjusting the timeout retry window based on feedback from network conditions. These mechanisms together constitute an enhanced idempotent protocol. For example, in a test under an extreme county-level network environment simulating a packet loss rate of up to 30% and an average network jitter of 297ms, the protocol designed by this method is expected to achieve an end-to-end success rate (after automatic retries) of over 99.95% for the synchronization of medical and health record data (such as resident contract signing and chronic disease follow-up records), without any data duplication or loss. This far exceeds traditional TCP-based simple direct transmission or message queue applications without enhanced idempotence (the success rate may drop to 70% in such an environment).

[0037] The data synchronization method proposed in this method, based on hybrid timestamp, shard checksum and CDC optimization, and enhanced idempotent protocol, can be applied to medical and health information systems, especially in the field of health record information interoperability of county-level medical communities. It is expected to build an efficient, reliable, and consistent data exchange and sharing foundation for county-level medical communities, supporting the collaborative development of various businesses such as clinical diagnosis and treatment, public health, and health management.

[0038] Example 2 This embodiment provides a highly robust real-time data synchronization method that implements data synchronization between a data source and a data target through message middleware, including the following steps: S1: At the data source end, the transaction log of the source database is monitored in real time. In response to the triggering of a data change event, the corresponding change data is captured and pre-processed; S2: Generate a hybrid timestamp for the pre-processed change data, where the hybrid timestamp includes a physical timestamp and a logical sequence number; S3: Perform data sharding on the changed data that meets the preset sharding conditions after preprocessing to obtain a plurality of sharded data; calculate an incremental checksum for each of the sharded data and / or unsharded changed data; S4: Encapsulate the pre-processed change data together with the mixed timestamp, incremental checksum, and pre-assigned globally unique message ID into a message, and send the message to a message queue preset in the message middleware; S5: At the data target end, obtain a message from the message queue and perform an idempotency check based on the message ID in the message. If the idempotency check passes, execute step S6; otherwise, discard the message and obtain a new message. S6: Recalculate the incremental checksum of the message and compare it with the incremental checksum encapsulated in the message. If the comparison result meets the preset conditions, the check passes and step S7 is executed. For messages that have undergone data fragmentation, data reassembly is performed first and then step S7 is executed. Otherwise, the check fails, the message is discarded, and a new message is obtained. S7: Determine the latest version of the message according to the mixed timestamp, execute the corresponding data change event based on the latest version of the message, and complete the data synchronization; In step S1, the transaction log of the source database is monitored in real time by using CDC technology, and the change data is captured; The preprocessing includes at least: parsing and formatting the captured change data; In step S2, the logical sequence number is specifically a sequence number that increases monotonically within the same physical timestamp; First, the physical timestamp is generated, and then the logical sequence number is generated; if the newly generated physical timestamp is the same as the physical timestamp of the last changed data, the logical sequence number is incremented by a preset value; when the physical timestamp is updated, the corresponding logical sequence number is reset to a preset initial value; In step S7, the physical timestamps of the messages are preferentially compared, and the message with the latest physical timestamp is determined as the latest version of the message; if the physical timestamps are the same, the logical sequence numbers of the messages are further compared, and the message with the largest logical sequence number is determined as the latest version of the message; In step S3, the preset fragmentation conditions include: the pre-processed change data is large field data of a preset type and / or batch change data; The data sharding process includes: dividing the pre-processed change data into a plurality of fixed-size shard data; or hierarchically dividing the pre-processed change data based on the data logical structure to obtain a plurality of shard data at different levels; In step S3, the incremental checksum is calculated using any one of the xxHash algorithm, CRC32 algorithm, MD5 algorithm, SHA-1 algorithm, and Rabin fingerprint algorithm; After receiving the message sent by the data source, the message middleware returns an ACK confirmation signal to the data source. If the message fails to be sent or the ACK confirmation signal is not returned due to timeout, the data source resends the message. After executing the corresponding data change event, the data target sends an ACK confirmation signal to the message middleware; The message middleware is provided with a dead letter queue for storing discarded messages and messages that have reached a preset maximum number of sending times; In step S6, a state library that can be updated in real time is provided in the data target end, and the state library stores at least the message ID of the message corresponding to the executed data change event; If the message ID of the current message already exists in the state library, the idempotency check fails; otherwise, the idempotency check passes.

[0039] In the specific implementation process, this embodiment still takes the health record data between county medical institutions as an example for explanation. Figure 2 The overall architecture of this method is shown in the figure. The specific steps are as follows: Step 1: Use a CDC component (such as Debezium) to monitor the transaction log of the specified source database in real time and capture raw data change events (including data before and after the change) generated by DML operations such as INSERT, UPDATE, and DELETE. In this step, use mature CDC tools such as Debezium and Canal, or leverage the database's logical replication capabilities to monitor the database's transaction log (such as MySQL Binlog and PostgreSQL WAL), capturing row-level DML operations (INSERT, UPDATE, and DELETE) and their pre- and post-change data images (for UPDATEs). Step 2, preprocessing and packaging: 1) Perform preliminary analysis on the captured change data to extract key information (table name, primary key, change type, and change data); 2) Generate a globally unique hybrid timestamp (Physical_TS, Logical_Seq) for the logical change; like Figure 3 As shown, the overall processing flow of the hybrid timestamp mechanism in this embodiment is generally divided into three parts: generation, update and conflict resolution of hybrid timestamps; the data structure of the hybrid timestamp consists of two parts, the data structure is (Physical_TS, Logical_Seq); Physical_TS represents a 64-bit high-precision physical timestamp, which records the server physical time when a data change event occurs, accurate to milliseconds or microseconds. For example, YYYY-MM-DD HH:MM:SS.sss (year-month-day-hour-minute-second-millisecond). This timestamp is derived from a reliable NTP synchronization server. Logical_Seq represents a logical sequence number, specifically a monotonically increasing integer sequence within the same physical timestamp (such as a millisecond). It consists of a 32-bit monotonically increasing sequence number and a 16-bit organization / device ID. For example, it can be an atomic counter or derived from the transaction ID / SCN (System Change Number) of the source database to ensure the unique order of multiple operations at the same time point. When CDC captures a data change (INSERT, UPDATE, DELETE), a hybrid timestamp is generated for it during the preprocessing phase. First, the current server's physical time is obtained as Physical_TS. Then, based on this Physical_TS, an incremental Logical_Seq is obtained or generated. If the current Physical_TS is the same as the Physical_TS of the previous record, the Logical_Seq is simply incremented by 1. If the Physical_TS changes (enters the next millisecond), the Logical_Seq is reset to its initial value (0 or 1). In addition, when CDC capture nodes are deployed in a distributed scenario, each node has its own predefined node id and can customize the priority of the node id. At this time, the generation of Logical_Seq uses a more complex coordination mechanism to ensure strict order within a single data source. Specifically, each node has its own independent sequence space and each node has its own predefined node id. In a normal scenario, Logical_Seq is an ascending sequence number of 0, 1, 2, 3, ... at the same timestamp, and the data order is 0>1>2> 3…; When there are multiple nodes, Logical_Seq adds a pre-defined node ID, and the structure becomes "node ID_node sequence number". For example, if the ID of node 1 is A001 and the ID of node 2 is B001, and the business scenario defines the priority A001>B001, then at the same timestamp, the Logical_Seq set (A001_0, A001_1, B001_0, B001_1) is generated, and the data order is A001_0>A001_1>B001_0>B001_1; When comparing two hybrid timestamps, the physical timestamp is prioritized. If the physical timestamps are identical (or the difference is within the tolerance range), the logical timestamps (such as the logical counter) are compared. If the logical timestamps are still identical, a final decision is made based on the predefined node ID priority. This hybrid approach balances the global comparability of physical time with the precise description of causal relationships by logical clocks. Step 3: Sharding and checksum calculation: 1) Determine whether the changed data is a large object or batch data. If so, split it into multiple data shards based on preset strategies (such as fixed size or data structure boundaries); 2) Calculate incremental checksums (e.g., using xxHash) for each data shard (or complete small data change). For UPDATE operations, optimize checksum calculations by leveraging the change information provided by CDC whenever possible. like Figure 4 As shown, in this embodiment, the sharding incremental checksum and CDC optimization combination mechanism mainly includes data sharding processing and incremental checksum calculation; In this embodiment, the data sharding strategy is primarily targeted at large field types (such as BLOB, CLOB, TEXT, JSON, and XML) in the database, or batch change data containing a large number of changes in a single transaction. Sharding is based on configurable fixed-size shards (such as one shard per 1MB) or on the logical structure of the data (such as a certain level of an XML / JSON document). After sharding, each shard carries shard information, including the unique identifier of the original data object, a shard sequence number (such as 1 / N, 2 / N, ..., N / N), and the total number of shards N. For medical data, this embodiment provides an example sharding strategy: sharding for patient basic information (including name, gender, date of birth, etc.), sharding for patient diagnosis history (including diagnosis name and time, etc.), sharding for patient allergy history (including allergens, reaction type, etc.), and sharding for patient medication history (including medication, dosage, and usage, etc.). Data sharding can effectively reduce the size of data packets transmitted over a single network, which is beneficial for improving the transmission success rate under unstable networks. It also facilitates parallel processing, refines the verification range, and quickly locates errors. For incremental checksum calculations, you can choose an efficient checksum algorithm, such as xxHash (fast), CRC32 (widely used), or MD5 / SHA-1 (high security requirements but computationally expensive). When choosing an algorithm, you should balance computational efficiency with collision probability based on your needs. When determining the "increment," a full checksum is calculated for INSERT operations or new shards. For UPDATE operations, CDC provides the changed fields and specific content. If the change is a local modification within a large object, the checksum is recalculated only for the affected block or region, combined with the "changed area" information provided by CDC, and quickly merged with the old checksum information for the unchanged part. If the shard granularity is small enough, the checksum can also be recalculated directly for the entire shard after the change. In this case, the "increment" is reflected in processing only the changed shard, rather than the entire large object. In this embodiment, "checksum" refers to the result of accumulating the checksum data and discarding the overflowed bits, resulting in one or more bytes. In simple terms, it is the calculation of the fingerprint of the data. The fingerprint is unique. When the data changes, the fingerprint will be inconsistent, which can be used to confirm whether the affected area is completely consistent with the changed area. In the incremental calculation of large objects, this method will divide the large object into blocks. Before data synchronization, the checksums of each block of the original object are compared in sequence to see if there are changes. The blocks with inconsistent checksums can be replaced to complete the incremental merge. For example, for a large field change object A, it is first divided into blocks (a1, a2, a3, ..., an), and then the checksums of each block of the original content and the new content are calculated proto(checkSum1, checkSum2, ..., checkSumn), new(checkSum1, checkSum2, ..., checkSumn). The checksums of each block are then compared to see if they are inconsistent. If they are inconsistent, it is confirmed that the block has changed, the actual content of the block is obtained from the change source and replaced in the target library to complete the change. In this embodiment, the transaction information captured by CDC can be used to define the boundaries of a batch of changes. This batch of changes can then be sharded and checksummed as a logical unit. The data images before and after the change provided by CDC facilitate accurate difference calculation and provide a basis for incremental checksums. Obtaining the precise scope of the change (e.g., which fields were modified) through CDC allows for more focused checksum calculations, avoiding unnecessary processing of unchanged data. This improves the efficiency of difference identification and data verification, especially when processing fine-grained data that is updated frequently. Step 4: Message construction and sending: 1) Encapsulate a message in a standard format (such as JSON) containing a mixed timestamp, shard information (such as shard ID, current shard sequence number, total number of shards, original object ID), incremental checksum, actual data (or data shard content), and a globally unique business ID or message ID (for idempotent processing); 2) The message producer securely sends the encapsulated message to the designated topic of the message queue through a reliable delivery mechanism (configured with ACK confirmation and retry); This embodiment not only relies on the basic guarantee of the message queue itself, but also improves robustness by introducing an application layer enhancement mechanism to ensure that data is efficiently and reliably transmitted and correctly processed between producers (data source end) and consumers (data target end) even in the case of network instability or node failure. The specific business process is as follows Figure 5 As shown; This example uses a message queue with high reliability, high throughput, and persistence, such as Apache Kafka or Pulsar, as the middleware. Message persistence to disk is enabled. Multiple partitions and replicas (Replication Factor > 1) are set for the topic to achieve high availability and fault tolerance. The producer configures "acks=all" (or a corresponding level) to ensure that messages are acknowledged by all ISRs (In-Sync Replicas). Consumers manually submit ACKs. The reliable transmission mechanism introduced by this method is as follows: a) Producer (data source): After sending a message, it waits for the middleware's ACK confirmation. If the message fails or times out, it retries (the number of retries and intervals can be configured, such as exponential backoff). b) Consumer (data destination): After pulling a message, it first performs business processing. Only when the business processing is successful and the data is successfully applied to the destination does it send an ACK confirmation to the middleware to inform the middleware that the message has been successfully consumed. If the application crashes during the consumption process, the unacknowledged message will be redelivered by the middleware to another (or the same) consumer after a timeout. c) The middleware is equipped with a dead letter queue (DLQ). Messages that continuously fail to be consumed due to reasons such as data format errors or inability to process by business logic are routed to a dedicated dead letter queue after the maximum number of retries has been reached. This avoids blocking normal message processing and allows for subsequent manual investigation. Step 5: Target-side message consumption and idempotence check: 1) The target consumer subscribes to and pulls messages from the message queue; 2) Based on the unique business ID / message ID in the message, the "Processed ID Status Database" is queried for idempotence. If the ID already exists, the message is discarded and an ACK is issued, terminating the process. Specifically, this embodiment assigns a globally unique ID (GUID / UUID, or based on the source transaction ID + sequence number) to each logical business change initiated from the source (for example, a complete patient information modification should have a common business transaction ID even if it is fragmented into multiple messages) or each independently sent message. This ID is sent along with the message. Before the target consumer receives the message and prepares to apply the change, it first queries a "processed ID state library" based on the unique ID in the message. The state library can be a cache (such as a Redis Set / Hash, which stores successfully processed IDs and can set an expiration time) or a persistent database table. If the ID already exists in the state library, indicating that the message (or business operation) has been processed, the current message is directly discarded (or only an ACK is returned) and no subsequent data application is performed, thereby achieving idempotence. If the ID does not exist, the data is processed normally and, after the data is successfully applied to the target system, the ID is stored in the state library. Furthermore, for business operations that require multiple shard messages to complete (e.g., all shards of a large file), the data destination must first cache the shards that have already arrived. Only after receiving the shard that marks the end or all expected shards have arrived and passed verification will the entire business operation be atomically applied to the target system, and the unique ID of the aggregated business will be recorded once. If a failure occurs midway, compensation or rollback is performed, or the operation is directly marked as failed and moved to a dead letter queue. This protocol design ensures that data will not be processed repeatedly or partially even in abnormal situations such as network retransmissions and consumer restarts, thus ensuring ultimate data consistency. Step 6: Data verification and reassembly: 1) For messages that pass the idempotence check, extract the data fragments (or complete data) and checksums; recalculate the checksum of the received data and compare it with the original checksum carried in the message; 2) If the verification fails, an error is logged and a limited number of retransmission requests may be attempted (if supported by the protocol) or the message may be moved to a dead letter queue; 3) If the message is a fragmented message, temporarily store the fragments that have passed the verification and check whether all fragments of the original data object have arrived. If not, wait; if all have arrived, reassemble all fragments into complete data according to the fragment sequence number; Step 7: Data application and conflict resolution: 1) For data that passes verification and is complete (non-sharded data or successfully reassembled sharded data), the mixed timestamp in the message is compared with the existing timestamp of the same record that may exist in the target system to determine the newness of the version; 2) Handle concurrent update conflicts according to a preset conflict resolution strategy. In this embodiment, the conflict resolution strategy can be "latest-come-first" or a more complex merge strategy based on business logic. For example, during the merge, nodes can be prioritized. For example, provincial-level institution nodes have a higher priority than municipal-level institution nodes. Among institution nodes at the same level, institution nodes with higher medical expertise in a certain disease type have a higher priority in data synchronization for that disease type. When a data change occurs where the priority cannot be determined, the data owner or the highest-level institution determines which data change to adopt. 3) Apply the final confirmed change data to the target database or system and perform the corresponding INSERT, UPDATE, and DELETE operations; Step 8: ACK confirmation and exception handling: 1) After the data is successfully applied to the target system, the consumer sends an ACK confirmation to the message queue, indicating that the message has been successfully processed; 2) If any error that cannot be automatically recovered occurs during the data application process (such as database constraint violation or business logic error), a detailed error log is recorded and the message (or its metadata) is moved to the dead letter queue for manual analysis and processing. At the same time, no success ACK is sent so that the message can be redelivered after a timeout.

[0040] The data synchronization method proposed in this embodiment is expected to produce the following significant positive effects in the medical and health information system, especially in the field of health record information interoperability within county-level medical communities: 1) Significantly reduce data synchronization latency and improve business response speed: Through real-time CDC capture, lightweight incremental processing (hybrid timestamps, sharding, incremental checksums), and an efficient asynchronous message transmission mechanism, this method can significantly reduce the average synchronization latency of key data such as health records and medical records between medical institutions within a county-level medical community from 15-30 minutes or even hours in existing solutions to less than 10 seconds. This near-real-time synchronization capability will greatly improve the service quality and efficiency of business scenarios with extremely high data timeliness requirements, such as clinical diagnosis and treatment (such as rapid sharing of emergency information and instant data support for remote consultations), public health emergency response, and dynamic health management. The availability of data updates is expected to increase by more than 95%. 2) Significantly improve data consistency across heterogeneous systems and ensure data quality: This solution ensures data integrity and accuracy at every stage of data transmission through a sharded incremental checksum mechanism. Combined with the precise version control capabilities provided by hybrid timestamps and a carefully designed conflict resolution strategy, as well as the elimination of duplicate data applications through idempotent processing protocols, this solution effectively resolves data inconsistencies caused by differences in data models, standards, and processing logic across heterogeneous systems (such as HIS, LIS, PACS, public health systems, and physical examination systems from different vendors). It is expected that the inconsistency rate of synchronized core health record data across different systems will be reduced from the historical average of 9% to below 0.1%, providing a solid and reliable data foundation for high-quality integration, in-depth analysis, and precise decision-making of regional medical data. 3) Significantly enhance the reliability and robustness of data synchronization in unstable network environments: In response to the common problems of narrow bandwidth, large network jitter, and easy interruption in county-level network infrastructure, this method designs a reliable transmission protocol based on message queues (including producer confirmation, consumer manual ACK, persistence, replication, and orderliness assurance) and an application-layer enhanced idempotent processing protocol, which gives the data synchronization process extremely strong fault tolerance and self-recovery capabilities. In an extreme test environment with a simulated network packet loss rate of up to 30% and an average network jitter of approximately 297ms, the final data synchronization success rate of this method (after a built-in retry mechanism) can still be maintained at above 99.95%. Compared with the traditional synchronization method based on TCP direct connection and lacking application-layer guarantees (the success rate may drop sharply to below 70% in such an environment), the reliability of this method has been improved by nearly 30 percentage points, effectively ensuring the continuity of various businesses within the medical community and the integrity of data without loss. 4) Reduced system operation and maintenance costs and complexity: Due to the highly automated, low-error, and fault-tolerant synchronization process, the need for manual intervention, troubleshooting time, and data verification complexity are expected to be significantly reduced, thereby lowering overall operation and maintenance costs. 5) Optimize system resource utilization: Through refined incremental synchronization, sharding, and optimized checksum calculation, network bandwidth usage can be reduced compared to traditional full or extensive incremental synchronization, thereby lowering the CPU and I / O load on both the source and target systems. 6) Improve system scalability and business flexibility: The loosely coupled, asynchronous architecture based on message queues makes the system easier to scale horizontally to cope with future growth in data volume and business types; at the same time, standardized interfaces and modular designs also facilitate rapid access to new data source systems or data consumption applications, thereby enhancing the flexibility and evolution capabilities of the entire medical community information platform.

[0041] Example 3 like Figure 6As shown, this embodiment provides a highly robust real-time data synchronization system, including a data source end 30, a message middleware 31, and a data target end 32. The data source end 30 includes the following modules: Change capture module 301: used to monitor the transaction log of the source database in real time, and in response to the triggering of data change events, capture the corresponding change data and perform pre-processing; Message construction and sending module 302: used to generate a hybrid timestamp for the pre-processed change data; perform data sharding on the pre-processed change data that meets the preset sharding conditions to obtain a plurality of sharded data; and calculate an incremental checksum for each sharded data and / or unsharded change data; and encapsulating the pre-processed change data together with the mixed timestamp, incremental checksum, and pre-assigned globally unique message ID into a message, and sending the message to a message queue preset in the message middleware 31; The data target end 32 includes the following modules: Idempotence check module 321: used to obtain a message from the message queue and perform an idempotence check based on the message ID in the message; Checksum and comparison module 322: used to recalculate the incremental checksum of the message and compare it with the incremental checksum encapsulated in the message; The data synchronization module 323 is configured to determine the latest version of the message according to the mixed timestamp, execute the corresponding data change event based on the latest version of the message, and complete data synchronization.

[0042] In the specific implementation process, the change capture module 301 monitors the transaction log of the source database in real time, and in response to the triggering of data change events, captures the corresponding change data and performs pre-processing; The message construction and sending module 302 generates a hybrid timestamp for the pre-processed change data; performs data sharding on the pre-processed change data that meets the preset sharding conditions to obtain a plurality of sharded data; calculates an incremental checksum for each sharded data and / or unsharded change data; and encapsulating the pre-processed change data together with the mixed timestamp, incremental checksum, and pre-assigned globally unique message ID into a message, and sending the message to a preset message queue in the message middleware 31; The idempotency check module 321 obtains a message from the message queue and performs an idempotency check based on the message ID in the message; The checksum comparison module 322 recalculates the incremental checksum of the message and compares it with the incremental checksum encapsulated in the message; The data synchronization module 323 determines the latest version of the message according to the mixed timestamp, executes the corresponding data change event based on the latest version of the message, and completes data synchronization.

[0043] The same or similar reference numerals correspond to the same or similar components; The terms used in the drawings to describe positional relationships are for illustrative purposes only and are not to be construed as limiting the present application. Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A highly robust real-time data synchronization method that uses message middleware to achieve data synchronization between a data source and a data target, characterized in that: The following steps are involved: S1: At the data source end, the transaction log of the source database is monitored in real time. In response to the triggering of a data change event, the corresponding change data is captured and pre-processed; S2: Generate a hybrid timestamp for the pre-processed change data, where the hybrid timestamp includes a physical timestamp and a logical sequence number; S3: Perform data sharding on the changed data that meets the preset sharding conditions after preprocessing to obtain a plurality of sharded data; calculate an incremental checksum for each of the sharded data and / or unsharded changed data; S4: Encapsulate the pre-processed change data together with the mixed timestamp, incremental checksum, and pre-assigned globally unique message ID into a message, and send the message to a message queue preset in the message middleware; S5: At the data target end, obtain a message from the message queue and perform an idempotency check based on the message ID in the message. If the idempotency check passes, execute step S6; otherwise, discard the message and obtain a new message. S6: recalculate the incremental checksum of the message and compare it with the incremental checksum encapsulated in the message. If the comparison result meets the preset conditions, the check passes and step S7 is executed. For messages that have undergone data fragmentation, data reorganization is performed first, and then step S7 is executed; Otherwise, the verification fails, the message is discarded, and a new message is obtained; S7: Determine the latest version of the message according to the mixed timestamp, execute a corresponding data change event based on the latest version of the message, and complete the data synchronization.

2. A highly robust real-time data synchronization method according to claim 1, characterized in that: In step S1, the transaction log of the source database is monitored in real time by using CDC technology, and the change data is captured; The preprocessing at least includes: parsing and format conversion of the captured change data.

3. A highly robust real-time data synchronization method according to claim 1, characterized in that: In step S2, the logical sequence number is specifically a sequence number that increases monotonically within the same physical timestamp; First, the physical timestamp is generated, and then the logical sequence number is generated; if the newly generated physical timestamp is the same as the physical timestamp of the last changed data, the logical sequence number is incremented by a preset value; When the physical timestamp is updated, the corresponding logical sequence number is reset to the preset initial value; In step S7, the physical timestamps between the messages are compared first, and the message with the latest physical timestamp is determined as the latest version of the message; if the physical timestamps are the same, the logical sequence numbers between the messages are further compared, and the message with the largest logical sequence number is determined as the latest version of the message.

4. A highly robust real-time data synchronization method according to claim 1, characterized in that: In step S3, the preset fragmentation conditions include: the pre-processed change data is large field data of a preset type and / or batch change data; The data sharding process includes: dividing the pre-processed change data into a number of fixed-size shard data; or, hierarchically dividing the pre-processed change data based on the data logical structure to obtain shard data of a number of different levels.

5. A highly robust real-time data synchronization method according to claim 1, characterized in that: In step S3, any one of the xxHash algorithm, CRC32 algorithm, MD5 algorithm, SHA-1 algorithm and Rabin fingerprint algorithm is used to calculate the incremental checksum.

6. A highly robust real-time data synchronization method according to claim 1, characterized in that: After receiving the message sent by the data source, the message middleware returns an ACK confirmation signal to the data source. If the message fails to be sent or the ACK confirmation signal is not returned due to timeout, the data source resends the message. After executing the corresponding data change event, the data target sends an ACK confirmation signal to the message middleware; The message middleware is provided with a dead letter queue for storing discarded messages and messages that have reached a preset maximum number of sending times.

7. A highly robust real-time data synchronization method according to claim 1, characterized in that: In step S6, a state library that can be updated in real time is provided in the data target end, and the state library stores at least the message ID of the message corresponding to the executed data change event; If the message ID of the current message already exists in the state library, the idempotency check fails; otherwise, the idempotency check passes.

8. A highly robust real-time data synchronization system, comprising a data source, a message middleware, and a data target, characterized in that: The data source includes the following modules: Change capture module: This module monitors the transaction logs of the source database in real time and, in response to data change events, captures the corresponding change data and performs preprocessing. Message building and sending module: used for generating a hybrid timestamp for the pre-processed change data; Perform data sharding on the changed data that meets the preset sharding conditions after preprocessing to obtain a plurality of sharded data; calculate an incremental checksum for each of the sharded data and / or unsharded changed data; and encapsulating the pre-processed change data together with the mixed timestamp, the incremental checksum, and the pre-assigned globally unique message ID into a message, and sending the message to a message queue preset in the message middleware; The data target end includes the following modules: Idempotence check module: used to obtain messages from the message queue and perform idempotence check based on the message ID in the message; Verification and comparison module: used to recalculate the incremental checksum of the message and compare it with the incremental checksum encapsulated in the message; Data synchronization module: used to determine the latest version of the message according to the mixed timestamp, execute the corresponding data change event based on the latest version of the message, and complete data synchronization.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are executed.

Images