Hierarchical verification-based data synchronization method, apparatus and device, and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a data synchronization method, device, equipment and medium based on hierarchical checking.The method comprises the steps that partitioning operation is conducted on source end data and target end data, a hierarchical check structure containing root check values is constructed, and the source end data and the target end data are synchronized; comparing whether the root verification values are consistent or not; and when the root check values are inconsistent, traversing and comparing the hierarchical check nodes, identifying data blocks which are different from the content of the target-end data block sequence in the source-end data block sequence, transmitting the different data blocks from the source-end device to the target-end device, and completing data updating at the target end. According to the method, accurate identification of data content differences is realized through a hierarchical verification structure, only data blocks with differences are transmitted, and waste of network and storage resources caused by a traditional full-quantity copying mode is avoided, so that the efficiency and reliability of data migration are remarkably improved on the premise of ensuring data consistency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method, device, equipment and storage medium based on layered verification. Background Art

[0002] In the context of the ongoing evolution of cloud computing architectures, the rapid growth of data volumes and the frequent migration of service instances have made data migration a core component of cloud platform operations and management. In typical scenarios, such as expiring computing resource replacements and data center infrastructure relocations, efficiently migrating data while ensuring data integrity and consistency has become a pressing technical challenge. This is particularly true in cloud platforms that rely on block storage as the underlying data carrier. Data migration tasks are not only massive in scale but also complex, posing numerous challenges, including high resource consumption and long migration cycles.

[0003] Taking block storage in mainstream cloud platforms as an example, existing data migration solutions typically use a full copy approach, transferring all data from the source device to the target device block by block. This approach is relatively simple to implement initially, but with the reality of ever-increasing data volumes, it has become increasingly problematic due to efficiency bottlenecks and resource waste. For block device data volumes of hundreds of GB or even TB, the full copy process can take hours or even longer, severely impacting business continuity and system maintainability. Furthermore, the indiscriminate transfer of all data during the migration process, including unchanged portions, consumes significant network bandwidth and storage resources, disrupting other system tasks.

[0004] In the fintech sector, critical systems such as core accounting databases and risk control model services are widely deployed on cloud hosts or database instances using block storage. These systems have extremely high requirements for data consistency and availability. However, when replacing resources or deploying disaster recovery in a remote location, the long downtime and additional resource consumption caused by traditional full migration methods can lead to transaction delays, user access failures, and even business interruptions, making it difficult to meet the demands of high-frequency changes and high stability.

[0005] In the healthcare sector, applications such as electronic medical record storage, imaging archiving, and chronic disease tracking and analysis also rely on block storage to carry core data. Data migration is a frequent occurrence in scenarios such as hospital information system upgrades and cross-regional cloud integration. Current full-copy migration solutions are not only time-consuming but also difficult to ensure timely completion in environments with limited network connectivity or short data access windows, impacting the continuity and responsiveness of medical services.

[0006] Furthermore, existing migration solutions generally lack mechanisms for analyzing data consistency between the source and target ends. This makes it impossible to quickly determine whether data has actually changed before migration, resulting in the repeated transmission of large amounts of redundant data. The lack of refined data verification and discrepancy identification mechanisms also necessitates additional measures to verify data consistency after migration, increasing management complexity and potential data risks. Summary of the Invention

[0007] The main purpose of the present invention is to provide a data synchronization method, device, equipment and storage medium based on hierarchical verification, aiming to solve the technical problem that the existing technology is unable to identify data differences between the source and target ends during data migration, resulting in full replication, low migration efficiency and serious waste of resources.

[0008] To achieve the above objectives, the present invention provides a data synchronization method based on hierarchical verification, comprising:

[0009] Performing data block operations on the source data carried by the source device and the target data carried by the target device to obtain a source data block sequence and a target data block sequence respectively;

[0010] Constructing a source-end hierarchical check structure including a source-end root check value based on the source-end data block sequence, and constructing a target-end hierarchical check structure including a target-end root check value based on the target-end data block sequence;

[0011] Comparing the source-end root checksum value with the target-end root checksum value;

[0012] When the source-end root check value is inconsistent with the target-end root check value, identifying a difference data block in the source-end data block sequence that has a different content from the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure;

[0013] The difference data blocks are transmitted from the source device to the target device, and the difference data blocks are received by the target device. The target data block sequence is updated based on the difference data blocks, and the target data is reconstructed.

[0014] Furthermore, to achieve the above-mentioned purpose, the present invention provides a data synchronization device based on hierarchical verification, comprising:

[0015] A data block module is used to perform data block operations on the source data carried by the source device and the target data carried by the target device, respectively, to obtain a source data block sequence and a target data block sequence;

[0016] a check structure construction module, configured to construct a source-side hierarchical check structure including a source-side root check value based on the source-side data block sequence, and to construct a target-side hierarchical check structure including a target-side root check value based on the target-side data block sequence;

[0017] A root checksum comparison module, configured to compare the source-end root checksum with the target-end root checksum;

[0018] a difference identification module configured to, when the source-end root check value is inconsistent with the target-end root check value, identify difference data blocks in the source-end data block sequence that have content differences with the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure;

[0019] A data synchronization module is used to transmit the difference data blocks from the source device to the target device, receive the difference data blocks through the target device, update the target data block sequence based on the difference data blocks, and reconstruct the target data.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data synchronization program based on hierarchical verification stored in the memory and runnable on the processor. When the data synchronization program based on hierarchical verification is executed by the processor, the steps of the data synchronization method based on hierarchical verification as described above are implemented.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data synchronization program based on hierarchical verification is stored. When the data synchronization program based on hierarchical verification is executed by a processor, the steps of the data synchronization method based on hierarchical verification as described above are implemented.

[0022] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A data synchronization method, device, equipment and medium based on hierarchical verification are disclosed, including: performing block operations on source-end data and target-end data, constructing a hierarchical verification structure containing a root verification value, and comparing whether the root verification values ​​are consistent; when the root verification values ​​are inconsistent, traversing and comparing hierarchical verification nodes, identifying data blocks in the source-end data block sequence that differ from the target-end data block sequence in content, transmitting the difference data blocks from the source-end device to the target-end device, and completing the data update at the target end. The present invention realizes accurate identification of data content differences through a hierarchical verification structure, and only transmits data blocks with differences, avoiding the waste of network and storage resources in the traditional full-volume copy method, thereby significantly improving the efficiency and reliability of data migration while ensuring data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0024] Figure 1 A schematic diagram of an application environment of a data synchronization method based on hierarchical verification in an embodiment of the present invention;

[0025] Figure 2 This is a flow chart of an embodiment of a data synchronization method based on hierarchical verification according to the present invention;

[0026] Figure 3 This is a functional module diagram of a preferred embodiment of a data synchronization device based on hierarchical verification according to the present invention;

[0027] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0028] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] The data synchronization method based on layered verification provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can perform block operations on the source terminal data and the target terminal data through the user terminal, construct a hierarchical verification structure including a root verification value, and compare whether the root verification values ​​are consistent; when the root verification values ​​are inconsistent, traverse and compare the hierarchical verification nodes, identify the data blocks in the source terminal data block sequence that have differences in content with the target terminal data block sequence, transfer the difference data blocks from the source terminal device to the target terminal device, and complete the data update on the target terminal. The present invention realizes accurate identification of data content differences through a hierarchical verification structure, and only transmits data blocks with differences, avoiding the waste of network and storage resources in the traditional full copy method, thereby significantly improving the efficiency and reliability of data migration while ensuring data consistency. Among them, the user terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 , Figure 2This is a flowchart of an embodiment of a data synchronization method based on hierarchical verification provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

[0032] like Figure 2 As shown, the data synchronization method based on layered verification proposed by the present invention includes the following steps:

[0033] S10, performing data block operations on the source data carried by the source device and the target data carried by the target device, respectively, to obtain a source data block sequence and a target data block sequence;

[0034] In this embodiment, the data block operation refers to dividing the complete data stored continuously or to be transmitted into a number of indivisible data units according to a preset size. These data units are called data blocks. In the source device, this operation can be applied to data entities such as logical volumes, partition images, database storage snapshots, and block-level storage snapshots. The division standard of the block operation can be a fixed-length division, for example, the size of each data block is 4MB, 8MB, or 64KB; or a dynamic threshold division method can be used to adjust the block granularity according to the data content, change frequency, or historical migration performance data. The data block process should ensure that each data block has a unique identifier to support the generation of subsequent verification structures and the identification of difference blocks.

[0035] The source device typically consists of elastic computing resources (such as ECS instances) or database service instances (such as MySQL or MongoDB) based on a virtualization platform. The source data carried on it refers to the business file system, database files, log data, or virtual disk image files on the device. The target device is the migration destination and can be located on the same physical network as the source or in a different data center. It is typically a newly deployed computing or database resource. The target data it carries is a reference copy formed by historical migrations or initial writes.

[0036] Block splitting is performed separately on the source and target sides. Each operation scans the original data content on their respective storage media and splits it according to a unified strategy. After the operation is completed, a source-side data block sequence and a target-side data block sequence are formed. A block sequence is a logical structure that records the order, logical offset, block size, and content summary of each data block, supporting subsequent data consistency checks and discrepancy detection. Block sequences can be stored in memory or cache using arrays, linked lists, or mapping tables. They can also be persisted to a database to support resumable downloads.

[0037] In actual deployment, data segmentation can be implemented based on the read interface of the block device. For storage devices that support standard block read and write protocols (such as SCSI and NVMe), data extraction and segmentation can be achieved through segmented reading. For example, the dd command in the Linux system can be called in combination with an offset parameter or implemented through programming based on the libaio asynchronous IO interface. To support the processing of large amounts of data, the system should be equipped with sufficient memory buffers to maintain data cache without loss during the segmentation process. RDMA technology can also be used for remote memory reading to achieve efficient block reading without occupying too many CPU resources.

[0038] In scenarios with complex data structures or high security requirements, the data block content can be masked, retaining only the content summary or encrypted content for subsequent operations to prevent sensitive data leakage. To store the block results, you can choose to build a temporary key-value database (such as LevelDB or RocksDB) to store the index and summary value of each block, facilitating fast comparison and difference queries.

[0039] If the target device has not yet initialized any data, you can remotely call the source's mirror snapshot interface to copy a structure skeleton as the starting point for the target data, and then perform block division based on the same strategy. If the target device already has migration data records, you can directly perform synchronous block calculation based on its current state.

[0040] Example: In the fintech business, when upgrading a customer credit assessment model or migrating a database to a new compute node, the existing customer profile database or transaction details table needs to be migrated from the old node to the new one. By partitioning the database storage volume at the source, the differences in data blocks between the old and new nodes can be quickly identified, avoiding full database migrations, reducing data synchronization time, and improving the efficiency of business system launches.

[0041] In the healthcare sector, when hospitals upgrade their electronic medical record system platforms or perform off-site backups, they need to migrate structured or unstructured data, such as imaging materials and medical records, from source storage nodes to the new platform. By adopting this data segmentation approach, only modified or newly added image blocks or document fragments need to be migrated to the target end, avoiding the repeated transmission of existing image files and ensuring continuous system availability and low-latency switching capabilities.

[0042] This embodiment performs independent data segmentation on both the source and target data, generating a logically structured sequence of data blocks. This abstracts the original, complete data into comparable basic units. This approach not only avoids the need for direct copying of full images but also lays the foundation for subsequent verification and difference analysis. This fine-grained data segmentation, achieved without changing the original data storage model, makes subsequent data synchronization more efficient, accurate, and resource-friendly.

[0043] S20, constructing a source-end hierarchical check structure including a source-end root check value based on the source-end data block sequence, and constructing a target-end hierarchical check structure including a target-end root check value based on the target-end data block sequence;

[0044] In this embodiment, the source-side data block sequence is a sequence of logical data units obtained by orderly partitioning the complete data on the source device. These units are typically divided into fixed-size units, such as 1MB or 4MB per data block, to facilitate subsequent structured processing and comparison. The data block partitioning logic should meet data alignment requirements while maintaining clear boundaries between data blocks to facilitate subsequent indexing and positioning in the tree structure. The target-side data block sequence is aligned with this sequence to ensure comparability of consistency judgments between the two devices in terms of physical or logical blocks.

[0045] The process of building a hierarchical verification structure refers to generating a hash structure system with multiple layers of nodes based on a sequence of data blocks. This system conforms to the tree topology characteristics in structure and is usually in the form of a Merkle tree. Each data block first generates a leaf node hash value through a one-way hash function (such as SHA-256, SM3, etc.), which serves as the bottom-level input of the tree structure. These hash values ​​are then merged in pairs according to a predetermined method, and the merged results are hashed again to generate the node value of the next layer. The hierarchical merging process will continue until a unique root node hash value is constructed, which is the root verification value. This value logically uniquely identifies the content status of the entire tree and all the data blocks at its bottom layer, and has tamper-resistance and consistency verification functions.

[0046] The source-side hierarchical checksum structure refers to the tree structure generated based on the source-side data block sequence, while the target-side hierarchical checksum structure is generated based on the target-side data block sequence. Although the two structures are identical, due to potential data differences, the hash values ​​of the nodes at each level may not match during comparison, especially at the intermediate or leaf node levels. The root checksum, as the top-level representation of the structure, is the starting point for consistency determination.

[0047] During the entire structure construction process, factors such as data alignment rules, hash function selection, node generation strategy, memory usage optimization strategy, and parallel hash calculation mechanism need to be considered. These factors will affect the efficiency of hierarchical verification structure construction and comparison accuracy.

[0048] A hierarchical parallel computing approach can be used to calculate initial hash values ​​for multiple data blocks. The hash results are then distributed to the structure building module through a thread pool scheduling mechanism for hierarchical node combination processing. During the node hash combination process, the maximum number of nodes to be merged per layer can be set, for example, to 2, 4, or 8, to achieve different granularities of tree structure depth control.

[0049] Hardware acceleration modules, such as cryptographic coprocessors with integrated SHA instructions, can be used to batch hash calculation tasks, thereby increasing build speed. Intermediate nodes in the calculation process can be stored in a buffer for subsequent comparison and difference location. To adapt to different target device platforms, the hash summary output format can be adjusted according to the word length supported by the target system during the build to ensure consistency with the storage structure.

[0050] A versioned hashing strategy can also be introduced to combine the node number of each layer with the data version number and then perform hashing, so that even if the position of the data block remains unchanged but the content changes slightly, the structural evolution path can be quickly identified.

[0051] Example: In the fintech business, a bank aims to minimize full data transmission to reduce network pressure when migrating block storage data between multiple active-active data centers. The system first performs data segmentation processing on the financial transaction log data in the source block storage, and then constructs a tree-like hierarchical verification structure on the source side based on the obtained transaction data blocks. This structure generates node hashes level by level based on the content of the transaction data blocks and forms a source root verification value at the top level. Similarly, the same data block segmentation and structure construction operations are performed on the existing data on the target side to form a target root verification value. This structured processing method allows the system to quickly compare the overall data status of both ends without relying on block-by-block comparison, which helps to accurately identify the financial data blocks that have actually changed and achieve low-overhead incremental migration.

[0052] In the healthcare business, a regional medical platform needs to synchronize patients' electronic medical records from local servers to the main cloud server. In actual operation, due to network bandwidth limitations and the large volume of medical record data, full synchronization is extremely inefficient. The system divides the medical record information files in the source server into structured blocks, and generates a source-side hierarchical verification structure based on the contents of these blocks. The root verification value generated at the top level is used to uniquely identify the overall data status of the current server. In the cloud, a hierarchical structure is also constructed based on the medical record data stored on the target server, and the root verification value is calculated. With this structured method, the time for comparing the medical record data status can be significantly shortened, and a structural foundation can be laid for the subsequent positioning of changed data, thereby achieving efficient and reliable medical data synchronization.

[0053] This embodiment, by constructing a hierarchical checksum structure including a root checksum value, can identify the consistency of data at both ends based solely on checksum comparisons, without transmitting the entire data set, significantly improving data synchronization efficiency. This hierarchical structure provides a structured path for locating difference data blocks, avoiding the complex computations associated with linear scans and enabling efficient incremental update determination.

[0054] S30, comparing the source-end root checksum value with the target-end root checksum value;

[0055] In this embodiment, after the data block structures at both ends are constructed, the root check values ​​generated by the source and target ends respectively constitute two representative hash digests. The key to the operation of this link is to compare the two root check values ​​at the binary level to determine whether the overall data status of the source and target ends is consistent. The root check value is usually obtained by merging the hash values ​​of the data blocks and intermediate nodes step by step. It has uniqueness and irreversibility, so its comparison result has strong representation. The specific comparison process can be completed based on the combination of hash value length consistency check and byte-by-byte content equivalence verification. This operation can be implemented using memory-level comparison instructions, for example, by performing byte-level displacement comparison, XOR judgment, etc. on the byte arrays occupied by the two check values ​​for verification. In the migration process where data integrity is extremely critical, this comparison process is the only structural operation that can perform rapid status judgment without relying on the data content itself.

[0056] During the comparison process, it is allowed to introduce checksum prefix compression processing, that is, only the first few bytes or bit segments are quickly judged. If a difference is found, the inconsistency is confirmed. Otherwise, the full comparison is completed byte by byte to improve performance. In addition, the comparison result of the checksum must have a clear Boolean output signal to facilitate the subsequent triggering of the difference path tracing logic. To prevent occasional comparison deviations, it is recommended to introduce a parallel comparison mechanism, that is, repeated calculation and comparison under a multi-threaded structure to ensure that the output result is stable and anti-interference during the calculation process.

[0057] In one implementation, the root checksum values ​​generated by the source and target are stored in two separate buffers in system memory. Content consistency verification is performed by writing a dedicated byte comparison function that calls the CPU instruction set's memcmp or a custom circular XOR algorithm. The comparison function immediately interrupts execution upon detecting any bit mismatch and returns a discrepancy signal. Alternatively, dedicated verification module hardware, such as an FPGA logic comparison engine, can be used to implement root checksum comparison operations for high-throughput data center scenarios, ensuring real-time responsiveness.

[0058] In a distributed architecture, the root checksums of the source and target can be compared and verified by a third-party controller, which can be deployed on a neutral scheduling node and is responsible for verifying data synchronization consistency. The comparison results are synchronized to the source and target via a signaling mechanism, allowing for a collaborative decision on whether to proceed with the discrepancy identification operation.

[0059] For example, in the fintech sector, a bank planned to migrate historical audit logs for multiple virtual hosts from the existing data center to a new one, hoping to avoid retransmitting large amounts of unchanged data. After structured log blocks and generating root checksums, the scheduling server compared the root checksums between the old and new data centers to quickly identify hosts whose log data was identical, allowing them to skip the migration. This comparison took only milliseconds, enabling a comprehensive assessment of the status of tens of thousands of bytes of data, significantly increasing the automation of the migration scheduling strategy.

[0060] In the healthcare sector, a regional hospital needs to periodically synchronize outpatient electronic records to a shared cloud platform. To account for cloud bandwidth limitations, the system generates a root checksum for the current outpatient data before each synchronization and compares it with the previously saved root checksum on the cloud. A difference update process is triggered only when the root checksum changes. This mechanism eliminates the need to repeatedly upload large numbers of unchanged records, effectively reducing network congestion and latency caused by redundant data transmission.

[0061] By comparing the root checksum values ​​of the source and target, this embodiment can efficiently determine whether the overall data status of both ends is consistent, without accessing all data blocks one by one. When the root checksum values ​​are consistent, redundant data synchronization processes can be skipped, significantly saving computing and bandwidth resources. When the root checksum values ​​are inconsistent, subsequent difference identification processes can be accurately guided, avoiding blind full data transmission. Therefore, this structured comparison method effectively realizes the rapid determination of data consistency during the migration process, improving overall migration efficiency and reliability.

[0062] S40, when the source-end root check value is inconsistent with the target-end root check value, identifying a difference data block in the source-end data block sequence that has a different content from that of the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure;

[0063] In this embodiment, when the root check values ​​obtained by independently constructing the data structures at both ends are inconsistent, it can be determined that there are actual differences between the source-end data and the target-end data. At this time, the complete data content should not be compared directly, but the source-end hierarchical check structure and the target-end hierarchical check structure should be compared layer by layer in a traversal manner. The hierarchical check structure is organized in a tree structure, in which each non-leaf node is generated by a combination of hash values ​​of its subordinate child nodes, forming a recursive hierarchical check system. During the traversal process, the hash value comparison of the check nodes at each level is performed pair by pair. If the hash values ​​of a pair of nodes are consistent, all their child nodes can be skipped, saving unnecessary comparison resources. When a hash value difference is detected between the corresponding check nodes at a certain level, it can be determined that there are data blocks with different contents in its subordinate child nodes.

[0064] To ensure the feasibility and accuracy of the comparison, each level node must contain the corresponding node index information and data block mapping location. In the process of traversal comparison, it is necessary to start from the root node, recursively carry out difference detection, and locate the leaf node layer by layer until the specific data block index location is finally identified. This process can synchronously record the corresponding mapping between the data block and the tree node when the data block sequence is initially generated, thereby achieving rapid positioning. Difference data blocks are defined as block-level units with any bit difference in content between the source data block and the target data block, not limited to the difference in the entire block, but also including changes at the tiny byte level.

[0065] A Merkle Tree construction tool can be used to recursively calculate the hash values ​​of data blocks on the source and target ends and generate their respective multi-level structures. During the comparison phase, a depth-first search strategy is preferably adopted, sequentially reading the hash values ​​of the corresponding check nodes at each level. For node pairs with inconsistent hash values, the search continues to the sub-node level; for node pairs with consistent hash values, all their sub-nodes are skipped. By maintaining a difference path index table, the paths are dynamically recorded during the traversal process, and ultimately a collection of indexes of all data blocks with differences is obtained. This collection can be used for subsequent transmission optimization, transferring only the data blocks that have actually changed to improve migration efficiency.

[0066] We can also leverage parallel traversal mechanisms to simultaneously traverse subtrees within a tree structure in a multi-threaded environment, further improving the efficiency of large-scale data structure comparisons. Furthermore, we can employ cache optimization strategies, utilizing intermediate hash value comparison results to avoid repeated computations and using a compressed storage structure for path results to reduce memory consumption.

[0067] Example: In healthcare business scenarios, patients' medical records, test results, and imaging reports are usually stored in electronic medical record systems. When a hospital migrates its data system, faced with the task of transmitting a large amount of continuous medical record data, it only needs to identify those data blocks that have changed due to new or modified records. By building a hierarchical verification structure and comparing the verification nodes of the source and target ends layer by layer, only the medical record data blocks containing difference information are located, thus avoiding the migration of large amounts of duplicate and unmodified data. This method based on differential data block positioning can significantly reduce bandwidth usage while ensuring the consistency of medical data and speed up the migration process. It is especially suitable for health monitoring systems with real-time update features.

[0068] In fintech business scenarios, distributed ledger platforms continuously record customer transaction behavior and fund changes. When migrating systems across regions, directly copying the entire transaction database is both time-consuming and risky. By traversing and comparing the hierarchical validation structures of the source and target ends, it is possible to quickly detect which transaction bill records or data blocks in the log files have been genuinely modified. For example, if only a few customers' balances have changed or new transactions have caused data discrepancies, the system can accurately identify the corresponding data blocks and instruct only these modified portions to be transferred subsequently, enabling refined ledger data migration and effectively reducing system synchronization latency and resource overhead. This approach meets the comprehensive real-time, consistency, and resource control requirements of financial services.

[0069] This embodiment uses a tree-like hierarchical verification structure to quickly and accurately locate the actual difference blocks in the data block sequence that have changed, avoiding redundant comparisons and full copying of the complete data. Compared to traditional byte-by-byte linear scan comparison methods, this method can significantly reduce the computational complexity and time overhead of the comparison operation, and significantly save transmission resources and bandwidth during the data migration process. By only identifying and transmitting the actual differences, overall migration efficiency is improved while ensuring data consistency.

[0070] S50 , transmitting the difference data block from the source device to the target device, receiving the difference data block through the target device, updating the target data block sequence based on the difference data block, and reconstructing the target data.

[0071] In this embodiment, the difference data block refers to a unit of data block content that is inconsistent after comparing the hierarchical verification structure of the source and target ends. It has the validity of the original data on the source end and is used to make up for the missing or distorted information content on the target end. The source device sends the difference data block to the target device through a network transmission mechanism. The transmission process usually requires the establishment of a channel with integrity verification capabilities, such as a data link that supports bidirectional data verification and error recovery mechanisms. After receiving the difference data block, the target device should provide a dedicated data receiving buffer area for temporarily storing the data units to be integrated to avoid data conflicts or structural damage caused by direct writing.

[0072] The received difference data block must be accurately mapped to the corresponding data block number or offset address in the target data block sequence in a logical position. This process is usually based on the position matching of the difference block identifier value generated by the source. After the matching is completed, the corresponding position in the target data block sequence is overwritten with the content of the newly received data block, forming an updated block sequence. The update operation must ensure write consistency and atomicity to avoid concurrency conflicts or partial write failures.

[0073] After the update is complete, the target data block sequence must be reconstructed to restore logically continuous data content. This reconstruction process relies on the logical order of the blocks and typically involves sorting the block numbers, concatenating the data segments, and forming a complete target data structure. This data structure can be a virtual volume mapped to a block device, a database file system image, or a collection of mirrored files.

[0074] During the transmission process, differential data blocks can be transmitted through an encrypted tunnel protocol, such as a TLS encrypted link that supports data compression and security verification, or through an RDMA communication channel with transmission reliability guarantees for zero-copy transmission. In the receiving stage, in order to improve transmission efficiency and write success rate, a memory-mapped buffer can be used to improve I / O throughput, and a write-ahead log can be configured to support fault rollback operations. When updating the target data block sequence, data writing can be performed in parallel through multi-threading to reduce latency; a transaction processing mechanism can also be used to ensure write consistency. During the data reconstruction process, different types of block structures can be adapted, such as file system blocks based on page alignment, database pages based on logical segments, or custom data snapshot formats.

[0075] Example: In fintech businesses, when core accounting systems or customer data service systems undergo data migration due to equipment warranty expiration or resource pool updates, data disks often spanning multiple terabytes. By comparing the root checksum values ​​on the source and target ends and identifying differential data blocks, only the actual changed accounting data blocks need to be transferred, such as certain high-frequency transaction records or customer points update fragments. During transmission, these differential data blocks are sent from the source device to the target device via a high-speed encrypted link. Upon receiving these data blocks, the target device immediately updates the original ledger block structure and reconstructs a complete and usable data volume image, enabling non-disruptive migration and rapid recovery.

[0076] In healthcare, for example, block storage systems used to store CT or MRI scan results in imaging systems typically only add a small number of new segments or modify some image annotation areas daily. When a hospital needs to copy the entire imaging system data due to equipment upgrades or cross-regional migration, traditional full-disk copies will result in redundant transmission. By identifying and transmitting differential data blocks, such as newly added scan layers or updated doctor annotation segments, the target device can accurately update the corresponding image segments and reconstruct the entire set of patient data after receiving the differential data blocks, ensuring that the medical system maintains the continuous availability of diagnosis and treatment data during the relocation process, avoiding duplicate examinations and treatment delays.

[0077] This embodiment significantly reduces network bandwidth and storage write pressure, shortening total transmission time, by transmitting differential data blocks instead of full data replication during large-scale data migration. On the target device side, data block sequences are updated based on differential data blocks, and the complete data content is further reconstructed, achieving end-to-end data consistency reconstruction capabilities and offering the application advantage of rapid migration across devices and networks.

[0078] The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A data synchronization method, device, equipment and medium based on hierarchical verification are disclosed, including: performing block operations on source-end data and target-end data, constructing a hierarchical verification structure including a root verification value, and comparing whether the root verification values ​​are consistent; when the root verification values ​​are inconsistent, traversing and comparing hierarchical verification nodes, identifying data blocks in the source-end data block sequence that differ from the target-end data block sequence in content, transmitting the difference data blocks from the source-end device to the target-end device, and completing the data update at the target end. The present invention achieves accurate identification of data content differences through a hierarchical verification structure, and only transmits data blocks with differences, avoiding the waste of network and storage resources in the traditional full-volume copy method, thereby significantly improving the efficiency and reliability of data migration while ensuring data consistency.

[0079] In one embodiment, the above step S10 includes:

[0080] S101, determining an initial block size according to a preset maximum block number threshold;

[0081] S102, based on the initial block size, determining the number of blocks of the source end data and the number of blocks of the target end data respectively;

[0082] S103, comparing the number of blocks of the source data with the maximum block number threshold, and if the number of blocks of the source data exceeds the maximum block number threshold, increasing the initial block size by a preset multiple to generate a source-side adjusted block size;

[0083] S104, comparing the number of blocks of the target end data with the maximum block number threshold, and if the number of blocks of the target end data exceeds the maximum block number threshold, increasing the initial block size by the preset multiple to generate a target end adjusted block size;

[0084] S105, dividing the source data into blocks based on the source-side adjusted block size to generate a source-side data block sequence;

[0085] S106 , dividing the target-side data into blocks based on the target-side adjusted block size to generate a target-side data block sequence.

[0086] In this embodiment, data stored in the source and target devices is partitioned to generate two sets of data block sequences. This process first requires determining a reference value for initial partitioning, and then setting a maximum partition threshold to limit the number of partitions. This maximum partition threshold is typically used to control the number of generated data blocks, avoiding excessive fragmentation or system resource overload when processing large amounts of data. The initial block size can be preset based on parameters such as data capacity, system processing power, and network transmission bandwidth. This block size directly affects the calculation of the number of blocks.

[0087] The data on the source and target sides are divided using the initial block size to determine the number of blocks for each. This calculation can be achieved by dividing the total amount of data by the initial block size and rounding up. If the number of divided blocks exceeds the maximum block number threshold, it means that the current blocks are too small and the total number of blocks needs to be reduced by increasing the initial block size. The increment operation uses a preset multiplication factor, usually a multiple of 2, 4, etc., to control the growth rate. The data structure and capacity of the source and target sides may differ, so the number of source and target block numbers needs to be threshold-matched and resized separately, and the source-side adjusted block size and target-side adjusted block size need to be independently generated.

[0088] After adjusting the block size, the data on both the source and target ends is partitioned according to the adjusted block size. Each partitioning step sequentially extracts the corresponding byte length from the starting address to generate data blocks until the entire data range is covered. This partitioning not only ensures a manageable number of blocks but also ensures the feasibility of subsequent block-based hierarchical verification structure construction, content comparison, and difference identification. By adjusting the block size rather than forcing uniform partitioning, the system avoids ignoring differences in the storage structure of the devices on both ends and enhances its adaptability in multi-environment deployments.

[0089] This embodiment effectively avoids the problem of data block surges caused by excessive data volumes through block number threshold control and dynamic block size adjustment, thereby reducing the system's burden on data management, indexing, and transmission. This mechanism allows the source and target ends to adjust their block strategies independently, allowing the system to maintain high block efficiency and structural consistency even when faced with heterogeneous storage environments or data structures with different capacities.

[0090] In one embodiment, the above step S20 includes:

[0091] S201, performing a one-way encrypted hash process on each data block of the source-end data block sequence and each data block of the target-end data block sequence, respectively, to generate a corresponding source-end leaf node hash value and a corresponding target-end leaf node hash value;

[0092] S202, arranging the hash values ​​of the leaf nodes on the source side in data block order as a bottom node set of a source-side hierarchical check structure, and arranging the hash values ​​of the leaf nodes on the target side in data block order as a bottom node set of a target-side hierarchical check structure;

[0093] S203, performing hierarchical merge hashing on the bottom node set of the source-end hierarchical checksum structure, generating source-end parent node hash values ​​layer by layer until a source-end root checksum value is generated;

[0094] S204 , performing hierarchical merge hash processing on the bottom node set of the target-end hierarchical check structure, generating target-end parent node hash values ​​layer by layer until a target-end root check value is generated.

[0095] In this embodiment, two multi-level data structures with complete verification capabilities are constructed to support subsequent data consistency verification and discrepancy location operations. This process requires constructing corresponding hierarchical verification structures based on the source and target data block sequences, respectively. Each structure ultimately outputs a root verification value that uniquely identifies the contents of the data block sequence.

[0096] First, a one-way cryptographic hash is performed on each data block. This process can utilize hash algorithms such as SHA-256 and SHA-512. The choice of different hash algorithms depends on the required security level and processing performance requirements. One-way cryptographic hashing is collision-resistant and reversible, making it suitable for data integrity verification. The hashing process is performed on each data block, and the output is the leaf node hash value. By performing this operation on the data block sequences on the source and target ends, two sets of leaf node hash values ​​are obtained.

[0097] Subsequently, the hash values ​​of these leaf nodes must be aligned with the order of the data blocks in the original sequence, forming a bottom-level node set. This sequential consistency of data blocks is key to maintaining the accuracy of subsequent hierarchical structure comparisons, avoiding misjudgments due to misaligned order.

[0098] Next, a hierarchical merge hashing process is performed on the aforementioned bottom-level node set. This process uses a layer-by-layer pairwise merge strategy, combining two adjacent hash values ​​and performing a hash calculation to generate the hash value of the parent node in the upper layer. If the number of nodes in the current layer is odd, the last node is copied to complete the pairing. This process is performed from bottom to top until only one node remains, which is the final root checksum. The data on the source and target sides are each constructed into an independent hierarchical structure, ultimately generating their own root checksum.

[0099] This structure can be considered a Merkle tree, featuring hierarchical traceability and locally verifiable nodes. The constructed source-side and target-side hierarchical verification structures can not only verify the full data content but also quickly locate data differences by comparing subtree paths, significantly improving the efficiency of data consistency checks and incremental updates.

[0100] This embodiment uses a hierarchical verification structure built based on a one-way encrypted hash algorithm to achieve efficient integrity verification of large-scale data block sequences and management of structured verification paths. This structure is not only scalable and traceable, but also can only update local paths when nodes change, thereby reducing the verification cost. The hash values ​​are arranged in the original order and the parent nodes are constructed layer by layer to ensure that the root verification value finally generated can fully reflect the content status of its corresponding data block sequence, providing a basic guarantee for subsequent high-precision difference identification. In this way, in scenarios such as data migration and synchronization, the risk of mismatch can be effectively reduced, and the stability and accuracy of the system in processing massive data consistency verification tasks can be improved.

[0101] In one embodiment, the above step S30 includes:

[0102] S301, extracting a source-end root check value from the source-end hierarchical check structure, and extracting a target-end root check value from the target-end hierarchical check structure;

[0103] S302, comparing the source-end root check value with the target-end root check value bit by bit;

[0104] S303: If the comparison results of all characters in the source-end root check value and the target-end root check value are consistent, determining that the source-end root check value is consistent with the target-end root check value;

[0105] S304: If the comparison result of at least one character between the source-end root check value and the target-end root check value is inconsistent, mark the source-end root check value and the target-end root check value as inconsistent.

[0106] In this embodiment, by comparing the root checksums generated by the source and target, we can quickly determine whether there are any differences in the overall data content. First, we need to extract the source root checksum and the target root checksum from the established hierarchical checksum structure. These two values ​​are located at the top-level nodes of their hierarchical structure and represent the overall cryptographic digest of their corresponding data block sequence. They are usually fixed-length hash strings, such as a 64-bit hexadecimal string (based on SHA-256) or a longer character sequence.

[0107] After extraction is complete, the two root checksums need to be compared character by character. This can be accomplished using byte-level circular comparison logic, comparing each character sequentially from the starting position. In high-performance environments, this can be converted into a byte array and then subjected to a bitwise XOR operation. If a character mismatch is detected at a specific position during the comparison, a discrepancy can be determined, terminating the comparison early without having to compare the entire character sequence, thus saving computational resources. If the comparison results for all characters are consistent, the two root checksums are considered equivalent, and the two data block sequences can be further deduced to be identical in content.

[0108] After the comparison is complete, if all characters match, the source and target root checksums are marked as consistent. This status can be used to skip data transfer or reconstruction operations in subsequent processes. If at least one character is inconsistent during the comparison, it should be marked as inconsistent, triggering the difference identification process to further locate the location of the different content.

[0109] This comparison logic relies on the accurate construction of the hierarchical structure and the consistent use of the hash function in the previous processing step. Therefore, in implementation, it is necessary to ensure that both parties use the same hash algorithm and encoding standards (such as hexadecimal encoding and consistent uppercase and lowercase) to avoid misjudgments due to encoding differences. In addition, to adapt to different scenarios, the character-by-character comparison can be optimized to a segmented parallel comparison strategy to improve the response efficiency of large-scale comparison tasks.

[0110] This embodiment uses character-by-character comparison based on root checksums to quickly determine overall data consistency without accessing the original data content, significantly reducing the overhead associated with block-level comparisons. When two root checksums are completely consistent, subsequent difference analysis steps can be skipped, saving computing and transmission resources. In the event of inconsistency, a simple comparison can quickly initiate the difference location process, implementing a hierarchical decision logic from coarse-grained verification to fine-grained analysis. This ensures verification accuracy while improving overall processing efficiency and system response speed.

[0111] In one embodiment, the above step S40 includes:

[0112] S401, when the source-end root check value is inconsistent with the target-end root check value, performing a hash value comparison on a current-level node set of the source-end hierarchical check structure and a current-level node set of the target-end hierarchical check structure, marking nodes in the current-level node set with inconsistent hash values ​​as an inconsistent node set;

[0113] S402, traverse all inconsistent nodes, and if the current node is a leaf node, mark the source data block corresponding to the current node as a difference data block;

[0114] S403: If the current node is a non-leaf node, obtain the child node set of the current node and use the child node set as a new current level node set;

[0115] S404, recursively performing hash value comparison, marking, and traversal operations until all inconsistent nodes are processed as leaf nodes;

[0116] S405: Aggregate the marked difference data blocks to generate a difference data block set.

[0117] In this embodiment, when the root checksum comparison results in an inconsistent state, the specific content difference cannot be determined. Therefore, further in-depth comparison of node information at corresponding levels in the two hierarchical checksum structures is required to locate the specific data block differences. First, processing begins at the top node of the hierarchical structure, extracting the node sets from the current level on both the source and target ends. Within these node sets, each node is hashed and merged with its next-level child nodes, and its corresponding hash digest value is stored.

[0118] A hash value comparison is performed for each pair of nodes in the current level. This comparison can be performed byte-by-byte or by pre-converting the nodes into numeric arrays for parallel comparison. If a hash value difference is found between a pair of nodes, it indicates that the corresponding substructures are inconsistent. These nodes are marked as inconsistent and added to the currently processed inconsistent node set. This set is used for subsequent recursive expansion operations.

[0119] When traversing each inconsistent node, we first need to determine whether the current node is a leaf node. If it is a leaf node, it means that the node directly corresponds to a data block, and its hash value is derived from the one-way encryption performed on the original data block content. Therefore, the data block corresponding to this leaf node is a difference data block and should be marked and included in the difference data block set.

[0120] If the current node is not a leaf node, it indicates that it contains multiple child nodes, each representing a hash combination result at a lower level. In this case, further expansion is required to extract all child nodes under the node as the input node set for the next round of comparison. The same hash value comparison, difference node marking, and traversal judgment operations are then performed on these child nodes as described above.

[0121] This process runs recursively, processing the children of all nodes in the current inconsistent node set in each recursive round until all paths ultimately terminate at leaf nodes. Once all inconsistent nodes have been recursively expanded and converted into difference data blocks, the processing phase ends and all marked difference data blocks are output, forming the final difference data block set.

[0122] During implementation, a tree-structured index mapping mechanism can be introduced. By maintaining a numbered mapping relationship between node paths in the source and target verification structures, hierarchical nodes can be quickly matched. Furthermore, to improve performance, hierarchical labels or path identifiers can be added during the tree-structured node construction phase, eliminating the need for path reconstruction or sequential search operations during recursive traversal. A parallel traversal strategy can also be employed, assigning the substructure comparison tasks of multiple inconsistent nodes to different processing threads, thereby improving overall recognition efficiency.

[0123] This embodiment, through a recursive traversal and node comparison mechanism, can accurately identify data blocks with content differences between the source and target data blocks, while only processing the hash values ​​in the hierarchical verification structure. This avoids repeated comparison and transmission of the entire data content. This approach establishes an efficient path from overall consistency judgment to local difference extraction. In large-scale data migration or synchronization tasks, only the actual differences need to be processed, thereby reducing data transmission overhead, improving comparison accuracy and response speed, and significantly improving operational efficiency in environments with limited storage and network resources.

[0124] In one embodiment, the above step S50 includes:

[0125] S501, performing compression processing on each difference data block in the difference data block set to generate a compressed data block;

[0126] S502, sending the compressed data block to the target device through the secure transmission channel of the source device;

[0127] S503, receiving a transmission data packet from the source device through the network interface of the target device, the transmission data packet including the compressed data block;

[0128] S504, storing the transmission data packet in the cache area of ​​the target end device to generate a temporary cache data block;

[0129] S505, performing decompression processing on the temporarily cached data block to restore it to the original difference data block;

[0130] S506, determining the position of the corresponding data block to be updated in the target-side data block sequence according to the position identifier of the original difference data block in the source-side data block sequence;

[0131] S507, writing the original difference data block to the location of the data block to be updated, overwriting the original data block content in the target end data block sequence;

[0132] S508, upon completion of writing, triggering a rebuild operation of the target-side hierarchical checksum structure to generate an updated target-side root checksum value;

[0133] S509, reconstructing the target-end data carried by the target-end device based on the updated content of the target-end data block sequence;

[0134] S510: Compare the updated target-end root check value with the source-end root check value. If they are inconsistent, re-request the corresponding difference data block.

[0135] In this embodiment, the differential data block transmission phase uses a differential data block set as input. Each data block in the set to be transmitted must be compressed to reduce the amount of data transmitted. This compression process can be based on a common lossless compression algorithm such as LZ4, Zstandard, or Snappy, ensuring that the recovered content is completely consistent after decompression. Each compressed data block is accompanied by a location identifier, which is used for subsequent location and update of the data block sequence on the target end.

[0136] The compressed data blocks are sent via a secure transmission channel established within the source device. This channel should support encrypted transmission protocols such as TLS, SSH, and IPSec to ensure that the data content is not tampered with or leaked during transmission. The source encapsulates each compressed data block into a network transmission packet. In addition to the data block content, the transmission packet should also contain metadata such as location identifiers and verification information.

[0137] The target device receives these packets through its network interface. Received packets are first buffered in the device's memory or dedicated cache storage area to form temporary cache blocks. This buffering mechanism allows for integrity checks and unpacking operations before decompression, thus ensuring data validity.

[0138] Temporarily compressed data blocks in the cache are decompressed. This decompression process must be consistent with and integrated with the source compression algorithm to ensure that the difference data blocks can be fully restored to their original pre-compression content. The decompressed data blocks retain their original location identifiers.

[0139] The target end then needs to identify the corresponding write location in the target end data block sequence based on the location identifier. This positioning process can be combined with a pre-generated data block index table or location information mapping table to ensure that the data overwrite operation does not accidentally write or cross the boundary.

[0140] The write operation is performed as an overwrite, writing the decompressed original difference data block content to the specified location in the target data block sequence, replacing the original inconsistent content. After the data is written, the target hierarchical checksum structure is rebuilt. This structure reconstructs the leaf node hash value from the updated data block, and generates the parent node hash value layer by layer until the target root checksum is updated.

[0141] After the target-side data block sequence is differentially updated, to ensure that the target device has fully consistent business data content with the source data, a target-side data reconstruction operation must be performed based on the updated data block sequence. This process first reconstructs the data blocks in their original order based on the order or position identifiers recorded during the block splitting phase, restoring a complete logical data volume. If the reconstruction involves file system or database structures, the restoration of special metadata must also be considered, such as file system superblock information, database file page headers, or transaction log blocks. This metadata typically has a specific structure format and location dependencies and must be inserted or combined according to predetermined rules to ensure the integrity and business availability of the final data. Furthermore, alignment strategies, such as block boundary alignment or page alignment, may have been employed during the block splitting process. During reconstruction, excess padding must be removed or the alignment restored. To verify the correctness of the reconstructed data, the target can perform integrity checks on the reconstructed data, such as end-to-end consistency checks based on hash functions or structural integrity verification using file system checking tools. Once the reconstruction is complete and verified, the target server maps the dataset to a local storage path or mount point, allowing it to continue operating as accessible business data, completing the entire business switchover process after differential synchronization. This process allows the target server to accurately restore the source data without transferring the entire data set, improving migration efficiency and ensuring data consistency and reliability.

[0142] After the data is overwritten and the checksum structure is rebuilt, the root checksum comparison between the source and target is performed again. This process is consistent with the aforementioned comparison mechanism and is used to confirm whether the current update has completely reconciled the data on both ends. If the comparison still results in inconsistencies, the difference data block identification process can be restarted and supplementary transfers can be performed until the root checksums on both ends are consistent.

[0143] This embodiment accurately updates the target data block content by performing a series of operations on the differential data blocks, including compression, transmission, caching, decompression, write positioning, and structural reconstruction, without having to transmit all the original data blocks. This process significantly reduces transmission bandwidth and processing resource consumption while ensuring data consistency. A closed-loop verification mechanism, implemented through final root checksum comparison, further enhances the reliability and integrity of the data migration process, making it effectively applicable to efficient incremental synchronization scenarios in resource-constrained or large-scale distributed storage environments.

[0144] In one embodiment, the above step S510 includes:

[0145] S5101: Input the updated target-end root checksum and the source-end root checksum into a consistency comparison module to generate a global checksum result.

[0146] S5102: If the global check result is inconsistent, analyze the root node association path of the target-side hierarchical check structure and the source-side hierarchical check structure to determine the impact range of the difference data block;

[0147] S5103: Generate a difference data block retransmission request according to the impact range of the difference data block;

[0148] S5104: Send the difference data block retransmission request to the source device through the secure transmission channel of the target device;

[0149] S5105, receiving an updated difference data block set returned by the source device based on the difference data block retransmission request;

[0150] S5106: Perform integrity check on the updated difference data block set. If the check passes, re-trigger the data writing and hierarchical check structure reconstruction operations.

[0151] In this embodiment, after completing the initial round of differential data block writing and rebuilding the target-side hierarchical verification structure, a consistency check of the root verification value is performed to finally confirm whether the target-side data is completely consistent with the source-side data. The updated root verification value of the target side and the root verification value of the source side are respectively input through the consistency comparison module. The module can perform a character-by-character comparison based on a hash algorithm or character-level verification logic to generate a global verification result to determine whether the two values ​​are consistent. This process usually does not rely on traditional database verification and comparison logic, but uses an efficient binary string comparison algorithm to quickly determine content consistency, ensuring that no additional computing resource overhead is introduced.

[0152] When the verification result indicates inconsistency, the system needs to determine whether there are any issues with write failures, decompression errors, or missing data packets during the new write process. At this point, the system extracts the path mapping information between the root nodes from the target-side and source-side hierarchical verification structures, including the parent-child node hash relationship and the original data block index mapping, to construct an associated path graph. By analyzing this path graph, the affected data range is precisely identified, thereby deriving the set of differential data blocks that may still contain anomalies.

[0153] Based on the above derivation, the system constructs a retransmission request data structure containing the position index of each block in the data block set. This request is encapsulated via TLS or another trusted communication protocol and sent from the target device to the source device. Upon receiving the retransmission request, the source device re-extracts the corresponding data block from the original source data block sequence, performs a new compression operation on it, and returns an updated set of difference data blocks.

[0154] After receiving the updated differential data block set, the target device does not write it immediately. Instead, it first runs a hash integrity check algorithm (for example, re-running the SHA-256 hash and comparing it with the previously generated value) to confirm that the data has not been damaged or tampered with during network transmission. If the check passes, the system restarts the write process, including overwriting the written data block, refreshing the cache, and triggering the reconstruction of the hierarchical checksum structure to ultimately form a new target root checksum value. This ensures that after multiple interactions, data consistency can be finally confirmed.

[0155] Example: In the financial sector, when a bank's core accounting system faces the need for off-site disaster recovery deployment, it often requires efficient synchronization of customer account master data and historical transaction flows in the general ledger system from the primary data center to the disaster recovery data center. This synchronization process begins by extracting the complete accounting database from the block storage of the original production node and extracting the existing base image data from the disaster recovery node, performing block partitioning on each node. Initially, a default block size (e.g., 4MB) is determined based on a preset maximum block number threshold (e.g., 8192 blocks). The total amount of data on the source and target nodes is then counted, and the current number of blocks is calculated. If the number of source-side blocks exceeds the threshold (e.g., 10,000 blocks), the initial block size is increased by multiples of 2 to 8MB to generate a new source-side block partition. Similarly, if the target-side block size also exceeds the threshold, the target-side block size is adjusted in the same manner. Block partitioning on both the source and target nodes is completed based on this adjusted size, forming a serialized data block set. Next, the SHA-256 cryptographic hash function is applied to each data block on both ends, generating a block-level hash value set. These are then used as leaf nodes in a Merkle tree (a hierarchical checksum structure) and arranged sequentially in the order of the original data blocks. Starting from the leaf nodes, the system concatenates the adjacent hash values ​​layer by layer in a binary tree structure and hashes them again to generate the upper-level node, and so on, until a unique root hash value is formed, which serves as the root checksum for both the source and target ends. The system extracts the root hash values ​​from the two trees and compares them character by character. If they match, the data content is identical, completing the fast synchronization checksum. However, if the comparison results are inconsistent, it indicates that the content of at least one set of data blocks has shifted or changed. At this point, the system automatically initiates the difference location process. By comparing the hash value sets of the nodes at the current level, it identifies the inconsistent node set and determines whether it is a leaf node or an intermediate node. If it is an intermediate node, its child nodes are retrieved and the comparison continues recursively downward. Ultimately, all recursive comparison processes reach the leaf node level, accurately locating all data differences and forming a set of different data blocks. After forming a set of differential data blocks, each data block is compressed, for example, using the Zstandard compression algorithm to reduce the network transmission load, and then sent to the disaster recovery node through the established TLS secure transmission channel. The target device receives the data packet through its network interface and stores it in the cache area, and then decompresses it to restore the original data block. Based on the block index information of each differential data block on the source side, the system accurately locates the corresponding position on the target side and performs content overwriting. After the write is completed, the system automatically rebuilds the Merkle tree structure on the target side and generates a new target root check value. Finally, the source and target root check values ​​are compared again. If there is still inconsistency, the unfinished differential block transmission operation can be triggered to ensure the final consistency completion confirmation.

[0156] In the healthcare sector, this system is suitable for data synchronization tasks during cloud migration or off-site disaster recovery deployments for hospital radiology archiving systems (PACS). For example, a city-level tertiary hospital's image server needs to migrate tens of terabytes of raw MRI and CT image files from local private servers to a provincial medical data platform for archiving and subsequent AI analysis. The system first partitions the initial datasets in the MRI image storage array and the target cloud platform at the block level, using a 6MB block size. This size is dynamically adjusted to 12MB or larger when the maximum number of blocks is exceeded. A hash function is then applied to each block to extract a unique identifier, building a hierarchical verification structure to quickly locate changes. For example, during the migration process, some images may have been updated due to doctors re-entering examination conclusions locally, resulting in a root hash mismatch between the source and target servers during verification. The system immediately triggers a discrepancy identification process, precisely identifying the modified image data blocks and compressing and transmitting them to the target platform. Through targeted overwriting based on location indexes, the system eliminates the need for repeated migrations and achieves efficient incremental updates at a block-level granularity. Finally, by rebuilding the checksum tree and verifying the updated hash consistency, the data stored on the new platform is ensured to be consistent with the local data, supporting subsequent AI-assisted diagnosis models to use the original image data to perform training or prediction tasks.

[0157] Through the above-mentioned method, this embodiment can implement a fault-tolerant mechanism that performs secondary verification after data transmission and writing, thereby effectively solving the problem of data inconsistency between the target and source ends caused by network fluctuations, cache write anomalies, or data packet damage. This mechanism determines the range of differences through path resolution and supports refined retransmission of difference blocks. Compared with the traditional full retry method, it greatly reduces the volume of retransmitted data and the time cost, and improves the stability and recoverability of the system during large-scale data synchronization. Ultimately, by constructing a multi-round verification feedback and local reconstruction mechanism, the robustness and consistency assurance capabilities of the overall data migration process are improved.

[0158] In one embodiment, a data synchronization device based on hierarchical verification is provided, and the data synchronization device based on hierarchical verification corresponds one-to-one to the data synchronization method based on hierarchical verification in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a data synchronization device based on hierarchical verification. It includes a data segmentation module 10, a verification structure construction module 20, a root verification value comparison module 30, a difference identification module 40, and a data synchronization module 50. Each functional module is described in detail below:

[0159] The data block module 10 is used to perform data block operations on the source data carried by the source device and the target data carried by the target device, respectively, to obtain a source data block sequence and a target data block sequence;

[0160] A check structure construction module 20 is configured to construct a source-side hierarchical check structure including a source-side root check value based on the source-side data block sequence, and to construct a target-side hierarchical check structure including a target-side root check value based on the target-side data block sequence;

[0161] A root checksum comparison module 30 is configured to compare the source-end root checksum with the target-end root checksum;

[0162] a difference identification module 40 configured to, when the source-end root check value is inconsistent with the target-end root check value, identify difference data blocks in the source-end data block sequence that have content differences with the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure;

[0163] The data synchronization module 50 is configured to transmit the difference data blocks from the source device to the target device, receive the difference data blocks through the target device, update the target data block sequence based on the difference data blocks, and reconstruct the target data.

[0164] In one embodiment, the data segmentation module 10 is specifically configured to:

[0165] Determine the initial block size based on a preset maximum block number threshold;

[0166] Based on the initial block size, determining the number of blocks of the source end data and the number of blocks of the target end data respectively;

[0167] Comparing the number of blocks of the source data with the maximum block number threshold, if the number of blocks of the source data exceeds the maximum block number threshold, increasing the initial block size by a preset multiple to generate a source-side adjusted block size;

[0168] Comparing the number of blocks of the target-end data with the maximum block number threshold, and if the number of blocks of the target-end data exceeds the maximum block number threshold, increasing the initial block size by the preset multiple to generate a target-end adjusted block size;

[0169] Dividing the source data into blocks based on the source end adjusting the block size to generate a source end data block sequence;

[0170] The target end data is divided into blocks based on the target end adjusting the block size to generate a target end data block sequence.

[0171] In one embodiment, the verification structure building module 20 is specifically configured to:

[0172] Performing one-way encrypted hash processing on each data block of the source-end data block sequence and each data block of the target-end data block sequence, respectively, to generate a corresponding source-end leaf node hash value and a corresponding target-end leaf node hash value;

[0173] Arrange the hash values ​​of the leaf nodes on the source side in data block order as a bottom node set of a source-side hierarchical check structure, and arrange the hash values ​​of the leaf nodes on the target side in data block order as a bottom node set of a target-side hierarchical check structure;

[0174] Performing a hierarchical merge hash process on the bottom node set of the source hierarchical checksum structure, generating source parent node hash values ​​layer by layer until a source root checksum is generated;

[0175] A hierarchical merge hash process is performed on the bottom node set of the target end hierarchical check structure, and a target end parent node hash value is generated layer by layer until a target end root check value is generated.

[0176] In one embodiment, the root check value comparison module 30 is specifically configured to:

[0177] Extracting a source-end root check value from the source-end hierarchical check structure, and extracting a target-end root check value from the target-end hierarchical check structure;

[0178] Comparing the source-end root checksum value with the target-end root checksum value bit by bit;

[0179] If the comparison results of all characters in the source-end root check value and the target-end root check value are consistent, then it is determined that the source-end root check value is consistent with the target-end root check value;

[0180] If the comparison result of at least one character between the source-end root check value and the target-end root check value is inconsistent, the source-end root check value and the target-end root check value are marked as inconsistent.

[0181] In one embodiment, the difference identification module 40 is specifically configured to:

[0182] When the source-end root check value is inconsistent with the target-end root check value, performing a hash value comparison on the current-level node set of the source-end hierarchical check structure and the current-level node set of the target-end hierarchical check structure, marking the nodes in the current-level node set with inconsistent hash values ​​as an inconsistent node set;

[0183] Traverse all inconsistent nodes. If the current node is a leaf node, mark the source data block corresponding to the current node as a difference data block.

[0184] If the current node is a non-leaf node, obtain the child node set of the current node and use the child node set as the new current level node set;

[0185] Recursively perform hash value comparison, marking, and traversal operations until all inconsistent nodes are processed as leaf nodes;

[0186] The marked difference data blocks are aggregated to generate a difference data block set.

[0187] In one embodiment, the data synchronization module 50 is specifically configured to:

[0188] Performing compression processing on each difference data block in the difference data block set to generate a compressed data block;

[0189] Sending the compressed data block to the target device through a secure transmission channel of the source device;

[0190] receiving a transmission data packet from the source device through a network interface of the target device, wherein the transmission data packet includes the compressed data block;

[0191] Storing the transmission data packet in the cache area of ​​the target end device to generate a temporary cache data block;

[0192] Decompressing the temporarily cached data block to restore it to the original difference data block;

[0193] Determine the position of the corresponding data block to be updated in the target end data block sequence according to the position identifier of the original difference data block in the source end data block sequence;

[0194] Writing the original difference data block into the position of the data block to be updated, overwriting the original data block content in the target end data block sequence;

[0195] When writing is completed, triggering a rebuild operation of the target-side hierarchical checksum structure to generate an updated target-side root checksum value;

[0196] reconstructing the target-end data carried by the target-end device based on the updated content of the target-end data block sequence;

[0197] The updated target end root check value is compared with the source end root check value. If they are inconsistent, the corresponding difference data block is requested again.

[0198] In one embodiment, the data synchronization module 50 is specifically configured to:

[0199] Inputting the updated target-end root checksum and the source-end root checksum into a consistency comparison module to generate a global checksum result;

[0200] If the global check result is inconsistent, analyzing the root node association path of the target end hierarchical check structure and the source end hierarchical check structure to determine the impact range of the difference data block;

[0201] generating a difference data block retransmission request according to an impact range of the difference data block;

[0202] Sending the difference data block retransmission request to the source device through the secure transmission channel of the target device;

[0203] Receiving an updated difference data block set returned by the source device based on the difference data block retransmission request;

[0204] An integrity check is performed on the update difference data block set, and if the check passes, the data writing and hierarchical check structure reconstruction operations are re-triggered.

[0205] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a data synchronization method based on hierarchical verification.

[0206] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data synchronization method based on hierarchical verification.

[0207] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0208] Performing data block operations on the source data carried by the source device and the target data carried by the target device to obtain a source data block sequence and a target data block sequence respectively;

[0209] Constructing a source-end hierarchical check structure including a source-end root check value based on the source-end data block sequence, and constructing a target-end hierarchical check structure including a target-end root check value based on the target-end data block sequence;

[0210] Comparing the source-end root checksum value with the target-end root checksum value;

[0211] When the source-end root check value is inconsistent with the target-end root check value, identifying a difference data block in the source-end data block sequence that has a different content from the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure;

[0212] The difference data blocks are transmitted from the source device to the target device, and the difference data blocks are received by the target device. The target data block sequence is updated based on the difference data blocks, and the target data is reconstructed.

[0213] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0214] Performing data block operations on the source data carried by the source device and the target data carried by the target device to obtain a source data block sequence and a target data block sequence respectively;

[0215] Constructing a source-end hierarchical check structure including a source-end root check value based on the source-end data block sequence, and constructing a target-end hierarchical check structure including a target-end root check value based on the target-end data block sequence;

[0216] Comparing the source-end root checksum value with the target-end root checksum value;

[0217] When the source-end root check value is inconsistent with the target-end root check value, identifying a difference data block in the source-end data block sequence that has a different content from the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure;

[0218] The difference data blocks are transmitted from the source device to the target device, and the difference data blocks are received by the target device. The target data block sequence is updated based on the difference data blocks, and the target data is reconstructed.

[0219] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0220] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0221] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0222] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data synchronization method based on hierarchical verification, characterized in that: The following steps are involved: Performing data block operations on the source data carried by the source device and the target data carried by the target device to obtain a source data block sequence and a target data block sequence respectively; Constructing a source-end hierarchical check structure including a source-end root check value based on the source-end data block sequence, and constructing a target-end hierarchical check structure including a target-end root check value based on the target-end data block sequence; Comparing the source-end root checksum value with the target-end root checksum value; When the source-end root check value is inconsistent with the target-end root check value, identifying a difference data block in the source-end data block sequence that has a different content from the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure; The difference data blocks are transmitted from the source device to the target device, and the difference data blocks are received by the target device. The target data block sequence is updated based on the difference data blocks, and the target data is reconstructed.

2. The data synchronization method based on hierarchical verification according to claim 1, characterized in that: Performing data block operations on the source data carried by the source device and the target data carried by the target device to obtain a source data block sequence and a target data block sequence, respectively, including: Determine the initial block size based on a preset maximum block number threshold; Based on the initial block size, determining the number of blocks of the source end data and the number of blocks of the target end data respectively; Comparing the number of blocks of the source data with the maximum block number threshold, if the number of blocks of the source data exceeds the maximum block number threshold, increasing the initial block size by a preset multiple to generate a source-side adjusted block size; Comparing the number of blocks of the target-end data with the maximum block number threshold, and if the number of blocks of the target-end data exceeds the maximum block number threshold, increasing the initial block size by the preset multiple to generate a target-end adjusted block size; Dividing the source data into blocks based on the source end adjusting the block size to generate a source end data block sequence; The target end data is divided into blocks based on the target end adjusting the block size to generate a target end data block sequence.

3. The data synchronization method based on hierarchical verification according to claim 1, characterized in that: Constructing a source-end hierarchical check structure including a source-end root check value based on the source-end data block sequence, and constructing a target-end hierarchical check structure including a target-end root check value based on the target-end data block sequence, including: Performing one-way encrypted hash processing on each data block of the source-end data block sequence and each data block of the target-end data block sequence, respectively, to generate a corresponding source-end leaf node hash value and a corresponding target-end leaf node hash value; Arrange the hash values ​​of the leaf nodes on the source side in data block order as a bottom node set of a source-side hierarchical check structure, and arrange the hash values ​​of the leaf nodes on the target side in data block order as a bottom node set of a target-side hierarchical check structure; Performing a hierarchical merge hash process on the bottom node set of the source hierarchical checksum structure, generating source parent node hash values ​​layer by layer until a source root checksum is generated; A hierarchical merge hash process is performed on the bottom node set of the target end hierarchical check structure, and a target end parent node hash value is generated layer by layer until a target end root check value is generated.

4. The data synchronization method based on hierarchical verification according to claim 1, wherein: Comparing the source-end root checksum with the target-end root checksum includes: Extracting a source-end root check value from the source-end hierarchical check structure, and extracting a target-end root check value from the target-end hierarchical check structure; Comparing the source-end root checksum value with the target-end root checksum value bit by bit; If the comparison results of all characters in the source-end root check value and the target-end root check value are consistent, then it is determined that the source-end root check value is consistent with the target-end root check value; If the comparison result of at least one character between the source-end root check value and the target-end root check value is inconsistent, the source-end root check value and the target-end root check value are marked as inconsistent.

5. The data synchronization method based on hierarchical verification according to claim 1, wherein: When the source-end root check value is inconsistent with the target-end root check value, identifying a difference data block in the source-end data block sequence that has a different content from the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure, including: When the source-end root check value is inconsistent with the target-end root check value, performing a hash value comparison on the current level node set of the source-end hierarchical check structure and the current level node set of the target-end hierarchical check structure, marking the nodes in the current level node set with inconsistent hash values ​​as inconsistent nodes; Traverse all inconsistent nodes. If the current node is a leaf node, mark the source data block corresponding to the current node as a difference data block. If the current node is a non-leaf node, obtain the child node set of the current node and use the child node set as the new current level node set; Recursively perform hash value comparison, marking, and traversal operations until all inconsistent nodes are processed as leaf nodes; The marked difference data blocks are aggregated to generate a difference data block set.

6. The data synchronization method based on hierarchical verification according to claim 1, characterized in that: The method includes transmitting the difference data block from the source device to the target device, receiving the difference data block by the target device, updating the target data block sequence based on the difference data block, and reconstructing the target data, including: Performing compression processing on each difference data block in the difference data block set to generate a compressed data block; Sending the compressed data block to the target device through a secure transmission channel of the source device; receiving a transmission data packet from the source device through a network interface of the target device, wherein the transmission data packet includes the compressed data block; Storing the transmission data packet in the cache area of ​​the target end device to generate a temporary cache data block; Decompressing the temporarily cached data block to restore it to the original difference data block; Determine the position of the corresponding data block to be updated in the target end data block sequence according to the position identifier of the original difference data block in the source end data block sequence; Writing the original difference data block into the position of the data block to be updated, overwriting the original data block content in the target end data block sequence; When writing is completed, triggering a rebuild operation of the target-side hierarchical checksum structure to generate an updated target-side root checksum value; reconstructing the target-end data carried by the target-end device based on the updated content of the target-end data block sequence; The updated target end root check value is compared with the source end root check value. If they are inconsistent, the corresponding difference data block is requested again.

7. The data synchronization method based on hierarchical verification according to claim 6, characterized in that: Comparing the updated target-end root checksum with the source-end root checksum, and if they are inconsistent, re-requesting the corresponding difference data block, including: Inputting the updated target-end root checksum and the source-end root checksum into a consistency comparison module to generate a global checksum result; If the global check result is inconsistent, analyzing the root node association path of the target end hierarchical check structure and the source end hierarchical check structure to determine the impact range of the difference data block; generating a difference data block retransmission request according to an impact range of the difference data block; Sending the difference data block retransmission request to the source device through the secure transmission channel of the target device; Receiving an updated difference data block set returned by the source device based on the difference data block retransmission request; An integrity check is performed on the update difference data block set, and if the check passes, the data writing and hierarchical check structure reconstruction operations are re-triggered.

8. A data synchronization device based on hierarchical verification, characterized in that: The data synchronization device based on layered verification includes: A data block module is used to perform data block operations on the source data carried by the source device and the target data carried by the target device, respectively, to obtain a source data block sequence and a target data block sequence; a check structure construction module, configured to construct a source-side hierarchical check structure including a source-side root check value based on the source-side data block sequence, and to construct a target-side hierarchical check structure including a target-side root check value based on the target-side data block sequence; A root checksum comparison module, configured to compare the source-end root checksum with the target-end root checksum; a difference identification module configured to, when the source-end root check value is inconsistent with the target-end root check value, identify difference data blocks in the source-end data block sequence that have content differences with the target-end data block sequence by traversing and comparing corresponding hierarchical check nodes of the source-end hierarchical check structure and the target-end hierarchical check structure; A data synchronization module is used to transmit the difference data blocks from the source device to the target device, receive the difference data blocks through the target device, update the target data block sequence based on the difference data blocks, and reconstruct the target data.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a hierarchical verification-based data synchronization program stored in the memory and capable of running on the processor. When the hierarchical verification-based data synchronization program is executed by the processor, the steps of the hierarchical verification-based data synchronization method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a data synchronization program based on hierarchical verification, and when the data synchronization program based on hierarchical verification is executed by the processor, the steps of the data synchronization method based on hierarchical verification according to any one of claims 1 to 7 are implemented.

Images