Non-intrusive heterogeneous database real-time synchronization system and method based on block chain

Abstract

The invention discloses a non-intrusive heterogeneous database real-time synchronization system and method based on a block chain, and belongs to the technical field of communication, and the system comprises a data storage module which is configured in a source database and is used for automatically monitoring the addition, deletion and modification operation of a target data table in real time, and capturing and storing changed data; the data synchronization module is configured outside the source database and is used for acquiring the to-be-synchronized change data from the data change storage module to push the change data to the target database and generating a synchronization receipt; and the block chain module is used for receiving and constructing the synchronous receipt generated by the data synchronization module, performing transaction verification on the constructed synchronous receipt through a consensus mechanism, and recording the transaction verification result in the block chain module. The system can solve the technical problem that the existing heterogeneous database synchronization technology lacks a transparent and credible auditing mechanism.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of communication, and particularly relates to a non-intrusive heterogeneous database real-time synchronization system and method based on a blockchain. BACKGROUND

[0002] Currently, multiple independently running business systems are usually deployed in an enterprise. Different business systems store data based on their core function requirements using differentiated database technologies, thereby forming a "data island" phenomenon. Therefore, breaking the "data island" and realizing data synchronization between heterogeneous databases has become a key technical requirement in the process of enterprise digital transformation.

[0003] At present, the mainstream technical solutions for heterogeneous database synchronization in the industry mainly include a batch synchronization solution based on a timing task and a synchronization solution based on a message queue. However, the above two solutions can solve the problem of heterogeneous database data synchronization, but both have the problem of lacking a transparent and credible audit mechanism in the data synchronization process of heterogeneous databases.

[0004] Therefore, there is an urgent need in the art for an innovative technical solution that can realize real-time synchronization of heterogeneous databases and has an unalterable, traceable, and highly credible audit capability throughout the synchronization process to solve the deficiencies of existing technical solutions.

[0005] In view of the above technical problems, the application provides a non-intrusive heterogeneous database real-time synchronization system and method based on a blockchain to solve the technical problem of lacking a transparent and credible audit mechanism in existing heterogeneous database synchronization technology. SUMMARY

[0006] In view of the above problems, the application provides a non-intrusive heterogeneous database real-time synchronization system and method based on a blockchain. The system comprises: a data storage module configured in a source database, used for automatically monitoring the addition, deletion, and modification operations of a target data table in real time, and capturing and storing changed data; a data synchronization module configured outside the source database, used for obtaining changed data to be synchronized from the data storage module to push to a target database, and generating a synchronization receipt; a blockchain module used for receiving and constructing the synchronization receipt generated by the data synchronization module, and performing transaction verification on the constructed synchronization receipt through a consensus mechanism and recording the constructed synchronization receipt in the blockchain module.

[0007] In the embodiment of the application, the data storage module comprises: a data change capture module configured to be implemented through a database trigger, used for automatically monitoring the addition, deletion, and modification operations of a target data table in real time, and capturing changed data; A data change storage module is configured to store the change data captured by the data change capturing module.

[0008] In the embodiments of the present application, the data synchronization module comprises a data synchronization table, and the data synchronization table comprises at least one of a table name, a self-incrementing ID, a primary key value, an operation type, pre-change data, post-change data, and an operation timestamp. The data synchronization table is configured to store the change data to be synchronized obtained from the data change storage module.

[0009] In the embodiments of the present application, the data synchronization module further comprises a data synchronization program, and the data synchronization program comprises: a data extraction unit configured to manage a synchronization point and extract the change data to be synchronized in batches from the data synchronization table; a data pushing unit configured to parse the change data to be synchronized and convert the change data to be synchronized into a data operation instruction in a format suitable for a target database, and push the data operation instruction to the target database; a synchronization receipt generation unit configured to generate a structured synchronization receipt after the data operation instruction is pushed to the target database.

[0010] In the embodiments of the present application, the synchronization receipt comprises at least one of a batch ID, an ID range of a processing record, a source database identifier, a target database identifier, a synchronization timestamp, execution node information, and an integrity hash value of batch data.

[0011] In the embodiments of the present application, the integrity hash value of the batch data is generated by constructing a Merkle tree, and the specific process comprises: using a hash algorithm on the change data in the same batch to generate a leaf node hash value; constructing a Merkle tree based on the leaf node hash value to generate a Merkle root hash value as the integrity hash value of the batch data.

[0012] In the embodiments of the present application, the blockchain module comprises: a blockchain service module configured to receive and construct the synchronization receipt generated by the data synchronization module, and sign and broadcast the constructed synchronization receipt by using a private key.

[0013] a blockchain network module configured to receive the constructed synchronization receipt broadcast by the blockchain service module, and verify the constructed synchronization receipt by a consensus mechanism and record the constructed synchronization receipt in the blockchain network module.

[0014] The present application further provides a non-intrusive heterogeneous database real-time synchronization method based on a blockchain, and the method comprises the following steps: using the data storage module configured in the source database to automatically monitor the add, delete and modify operations of the target data table in real time, and capture and store the change data. The data synchronization module configured outside the source database is used to obtain the change data to be synchronized from the data storage module to push to the target database, and generate a synchronization receipt; The blockchain module is used to receive and construct the synchronization receipt generated by the data synchronization module, and the constructed synchronization receipt is verified by a consensus mechanism and recorded in the blockchain module.

[0015] The application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the method in the above embodiment.

[0016] The application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the method in the above embodiment.

[0017] The system can solve the technical problem of lack of transparent and credible audit mechanism in existing heterogeneous database synchronization technology.

[0018] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by means of the structures particularly pointed out in the description and claims. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0020] Figure 1 A module schematic diagram of a blockchain-based non-intrusive heterogeneous database real-time synchronization system according to an embodiment of the present application is shown.

[0021] Figure 2 A blockchain-based non-intrusive heterogeneous database real-time synchronization process flowchart according to an embodiment of the present application is shown.

[0022] Figure 3 A blockchain-based non-intrusive heterogeneous database real-time synchronization method flowchart according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0024] In a modern information technology architecture, data has become a core strategic asset of an enterprise, and the circulation, sharing, and value-added capability of data directly determine the development efficiency and market competitiveness of the enterprise. With the continuous expansion of the scale of the enterprise and the continuous enrichment of the business scenarios, multiple independently running business systems are usually deployed in an enterprise. Different business systems store data based on their core functional requirements using differentiated database technologies, thereby forming a "data island" phenomenon.

[0025] For example, a system for processing online transaction business usually uses a relational database such as MySQL or Oracle to ensure the consistency and processing efficiency of transactional affairs. However, scenarios for data analysis, user portrait construction, or full-text search usually rely on a big data platform such as HBase, a search engine such as Elasticsearch, or a cache system such as Redis. The existence of the above-mentioned heterogeneous databases makes it difficult for data to flow efficiently between business systems, and seriously hinders the full release of data value.

[0026] Therefore, breaking the "data island" and realizing data synchronization between heterogeneous databases has become a key technical requirement in the process of enterprise digital transformation.

[0027] At present, the mainstream technical solutions for heterogeneous database synchronization in the industry mainly include the following two kinds: The first kind is a batch synchronization scheme based on a timing task: This scheme extracts target data from a source database according to a preset time period by writing a customized script, and loads the data to a target database after data format conversion. This scheme has the characteristics of mature technology and stable operation, and has been applied in some business scenarios with low requirements for synchronization real-time performance.

[0028] However, this scheme also has significant defects: first, the synchronization mode is T+1 level batch processing, with high synchronization delay, which cannot meet the needs of business scenarios with high real-time requirements; second, periodic batch reading operations on the source database will occupy a large amount of database resources, easily causing performance degradation of the source database and affecting the normal operation of core business.

[0029] The second kind: the synchronization scheme based on the message queue: the scheme requires the business system to actively encapsulate the data change information as a message and send it to the Kafka, RabbitMQ and other message queues after completing the database operation. The downstream target system obtains the data change information by subscribing to the corresponding message queue, and then completes the data synchronization. Compared with the batch synchronization scheme, the synchronization real-time performance is significantly improved.

[0030] However, this scheme has strong business invasiveness, and requires modification of the code of the existing business system, adding message sending related logic, which not only increases the complexity of system development and later maintenance, but also may introduce new system failure risks. At the same time, for the historical business system that has been put into operation, the modification is difficult, high in cost, and easy to cause business interruption during the modification process, and the application scope is greatly limited.

[0031] In summary, although the above two schemes can solve the problem of partial data synchronization, they both have the problem of lack of transparent and credible audit mechanism in the synchronization process.

[0032] Therefore, there is an urgent need in the art for an innovative technical solution that can achieve non-intrusive, efficient real-time synchronization of heterogeneous databases for business systems, and provide an unalterable, traceable and highly credible audit capability for the entire synchronization process, in order to solve the deficiencies of existing technical solutions.

[0033] In view of the above technical problems, the present application proposes a non-intrusive real-time heterogeneous database synchronization system and method based on blockchain, to solve the technical problems of strong invasiveness to business systems, poor heterogeneous compatibility and lack of transparent and credible audit means in the synchronization process in the existing heterogeneous database synchronization technology.

[0034] Figure 1 A module schematic diagram of a non-intrusive real-time heterogeneous database synchronization system based on blockchain according to an embodiment of the present application is shown, as shown in Figure 1 The system includes a data storage module, a data synchronization module and a blockchain module. The data storage module is configured in the source database, used to monitor the add, delete and modify operations of the target data table in real time, and capture and store the changed data. The data synchronization module is configured outside the source database, used to obtain the changed data to be synchronized from the data change storage module to push to the target database, and generate a synchronization receipt. The blockchain module is used to receive and construct the synchronization receipt generated by the data synchronization module, and through the consensus mechanism, the constructed synchronization receipt is verified and recorded in the blockchain module.

[0035] The blockchain-based non-intrusive heterogeneous database real-time synchronization system proposed in the application can receive and construct a synchronization reply generated by a data synchronization module, and through a consensus mechanism, the constructed synchronization reply is verified and recorded in the blockchain module through transaction, so as to solve the technical problem of lack of transparent and credible audit mechanism in the existing heterogeneous database synchronization technology.

[0036] Further, in the embodiment of the application, the data storage module includes a data change capture module and a data change storage module. The data change capture module is configured to be implemented through a database trigger, and is used to automatically monitor the add, delete and modify operations of the target data table in real time, and capture the changed data. The data change storage module is used to store the changed data captured by the data change capture module.

[0037] For example, in some embodiments, the blockchain-based non-intrusive heterogeneous database real-time synchronization system includes a source database, a target database, a data change capture module, a data synchronization module, and a blockchain module. The source database is a relational database for storing original business data. For example, the source database can be a MySQL, Oracle, or other database. The target database is a system or database that receives synchronization data. For example, the target database can be Elasticsearch, Redis, HBase, or other relational databases.

[0038] Taking the source database as a MySQL database and the target database as an Elasticsearch database as an example, in the embodiment of the application, the explanation and description are carried out. Specifically, the MySQL database includes at least one data table and a data synchronization table. For example, the data table is a "commodity information table", and the "commodity information table" can be synchronized to change information. When the "commodity information table" has data addition, deletion, or modification operation, the data change capture module based on the trigger on the source database is automatically activated to automatically monitor the add, delete, and modify operations of the target data table in real time, and capture the changed data.

[0039] Further, in the embodiment of the application, the data synchronization module includes a data synchronization table.

[0040] It should be noted that the data synchronization table is a log record table located in the source database, and is used to store the changed data to be synchronized obtained from the data change storage module. The data synchronization table includes at least one of a table name, an auto-increment ID, a primary key value, an operation type, pre-change data, post-change data, and an operation timestamp.

[0041] For example, the data synchronization table can include: table name, auto-increment ID, primary key value, operation type, pre-change data, post-change data. For another example, the data synchronization table can include: table name, auto-increment ID, primary key value, operation type, pre-change data, post-change data, operation timestamp. For another example, the data synchronization table includes: auto-increment ID, primary key value, operation type (add / INSERT, change / UPDATE, delete / DELETE), pre-change data (JSON format), post-change data (JSON format), operation timestamp.

[0042] Of course, in the embodiment of the application, the data synchronization module further includes a data synchronization program. The data synchronization program is the core execution unit of the system, and is mainly used for data synchronization and generating a synchronization receipt.

[0043] Further, the data synchronization program includes: a data extraction unit, a data pushing unit, and a synchronization receipt generation unit. The data extraction unit is used to manage synchronization points and batch extract change data to be synchronized from the data synchronization table. The data pushing unit is used to parse the change data to be synchronized and convert it into a data operation instruction in a format suitable for the target database, and push the data operation instruction to the target database. The synchronization receipt generation unit is used to generate a structured synchronization receipt after the data operation instruction is pushed to the target database.

[0044] It should be noted that in the embodiment of the application, the synchronization receipt includes at least one of: batch ID, ID range of processing records, source database identifier, target database identifier, synchronization timestamp, execution node information, and integrity hash value of batch data.

[0045] Preferably, in the embodiment of the application, the integrity hash value of the batch data is generated by constructing a Merkle tree, and the specific process includes: using a hash algorithm on the change data in the same batch to generate a leaf node hash value. A Merkle tree is constructed based on the leaf node hash value to generate a Merkle root hash value as the integrity hash value of the batch data.

[0046] Based on the above embodiment, in the embodiment of the application, the blockchain module includes: a blockchain service module and a blockchain network module. The blockchain service module is used to receive and construct the synchronization receipt generated by the data synchronization module, and sign and broadcast the constructed synchronization receipt using a private key.

[0047] The blockchain network module is used to receive the constructed synchronization receipt broadcast by the blockchain service module, and perform transaction verification on the constructed synchronization receipt through a consensus mechanism and record it in the blockchain network module.

[0048] Further, in the embodiments of the present application, the blockchain network module is configured as a consortium chain network module composed of multiple nodes based on Hyperledger Fabric or FISCO BCOS, wherein the consortium chain network module runs a smart contract in the network for recording and verifying the synchronous receipt.

[0049] Figure 2 A flowchart of a real-time synchronization process of a non-intrusive heterogeneous database based on a blockchain according to an embodiment of the present application is shown as Figure 2 As shown, for example, in one example, in a preset business scenario one, an operator modifies the price of a product with a primary key ID of "P001" from "99.00" to "129.00" in the "product information table" through the background management system. When the UPDATE operation is submitted to the source database, the data change capture module (i.e., the trigger) preset on the "product information table" is activated. The module captures the change data of this operation and inserts the change data to be synchronized obtained from the data change storage module into the data synchronization table within the same database transaction.

[0050] The content of the change data is as follows: ID: 2501 (auto-increment); table name: products; primary key value: P001; operation type: UPDATE; pre-change data: {"price": 99.00, "stock": 100}; post-change data: {"price": 129.00, "stock": 100}; operation timestamp: 2023-10-27 10:00:00.

[0051] At this time, the data synchronization program performs a synchronization task once every preset polling period (e.g., every 5 seconds). The data extraction unit inside the data synchronization program queries the recorded change data to be synchronized in a persistent storage (such as Redis), such as the "latest synchronized ID". Assuming that the current value of the "latest synchronized ID" is 2500.

[0052] The data extraction unit then initiates a query to the data synchronization table, requesting to obtain all change data records to be synchronized with an ID greater than 2500. In this scenario, the data extraction unit will obtain the record with an ID of 2501 and other possible newly added records. Of course, the data synchronization program can pull all change data records to be synchronized in batches. For example, 100 records are pulled at a time to improve work efficiency.

[0053] The data pushing unit receives the change data record to be synchronized with an ID of 2501 and parses the record to convert the general change data to be synchronized into a data operation instruction specific to the target database. This modularization of the data pushing unit can flexibly adapt to various heterogeneous databases.

[0054] In the embodiment of the present application, the target database is an Elasticsearch cluster, so the data pushing unit converts the operation record into an Elasticsearch DSL update request, the content of which is: update the document with _id P001 in the products index, and update the price field to 129.00.

[0055] If the target database is Redis or another relational database, only the conversion logic of the unit needs to be replaced to generate the corresponding command (such as the SET command of Redis or the UPDATE statement of SQL), without changing other links such as data capture and extraction. This design enables the entire synchronization system to support real-time synchronization to any heterogeneous database non-invasively. In addition, the data pushing unit sends the generated DSL to the target database (Elasticsearch cluster) through an HTTP request and waits for the response of successful execution.

[0056] It should be noted that after the change data to be synchronized in the above embodiment has been successfully pushed to the target database (in this case, for simplicity of description, it is assumed that only one piece of change data record with ID 2501 needs to be synchronized), the synchronization receipt generating unit is triggered and a structured "synchronization receipt" data is immediately generated. The synchronization receipt is a snapshot proof of this synchronization operation, and its content is as follows (represented in JSON format): {"batchId":"BATCH-20231027-005","processedIds":{"startId":2501,"endId":2501},"source":"MySQL-DB01","target":"Elasticsearch-Cluster01","syncTimestamp":"2023-10-2710:00:05","operatorNode":"SyncNode-A","dataHash":"a8c5d7e2f0..."} Among them, the "dataHash" field is a key credential for verifying the integrity of the batch of change data to be synchronized.

[0057] In the embodiment of the present application, in order to improve the computing efficiency and optimize the performance of batch processing of large-scale change data to be synchronized, the integrity hash value of the batch data is generated by constructing a Merkle tree, the process is as follows: Independent hash calculation (generate leaf nodes): First, the "changed data" (denoted as D1, D2, …, Dn) of each change data to be synchronized in this batch is independently applied to the SHA-256 hash algorithm, and the corresponding leaf node hash value h1, h2, …, hn is generated. hi = SHA-256(Di).

[0058] It should be noted that the tree structure aggregation (constructing a Merkle tree and generating a root): taking these leaf node hash values as the bottom layer, pairing and splicing them again and then performing hash calculation to generate the parent node of the upper layer. Repeat this process, aggregate layer by layer upwards, until a unique top hash value, i.e. the Merkle root, is finally generated. This Merkle root is the dataHash value written in the synchronization reply. The mathematical formula of the whole process is: dataHash = MerkleRoot(SHA-256(D1), SHA-256(D2), …, SHA-256(Dn)) The Merkle tree hash calculation has the following significant advantages: (1) Improve calculation efficiency: the initial hash calculation of each change data to be synchronized can be processed in parallel, which can greatly shorten the overall time consumption when processing a large amount of data in a multi-core CPU environment.

[0059] (2) Reduce memory consumption: avoid the memory overhead caused by concatenating all change data to be synchronized into a huge string. The main processing during the calculation process is the fixed-length hash value, and the process is more lightweight.

[0060] (3) Enhance auditability: provide efficient verification for subsequent audit tracing. When verifying whether a change data to be synchronized exists in the batch, the auditor does not need to obtain all the data in the batch, but only needs to provide the change data to be synchronized and its corresponding "Merkle path" (a hash chain branch from leaf to root) to complete the verification, which greatly improves the audit efficiency.

[0061] The data synchronization program sends the synchronization reply generated in the above embodiment to the blockchain service module through RPC or HTTP request. In the embodiments of the present application, the blockchain service module can be a chain agent service. After receiving the synchronization reply, the chain agent service calls the addSyncReceipt function in the smart contract pre-deployed on the blockchain network module. The function takes the JSON string of the synchronization reply as the function parameter, constructs a blockchain transaction, and signs it using its own digital certificate. Then, the signed transaction is broadcast to the blockchain network module.

[0062] The consensus node in the blockchain network module verifies, sorts and packages the transaction. Among them, the verification process includes checking the transaction signature, confirming that the caller (on-chain agent service) has the right to perform the operation, etc. After verification, the transaction is written into the new block in the blockchain network module and linked to the blockchain. At this point, the "synchronization receipt" of the data synchronization operation with ID 2501 is permanently and tamper-proof recorded. After the smart contract is successfully executed, a transaction hash (TxID) will be returned, which can be recorded by the on-chain agent service for future quick queries.

[0063] After confirming the success of on-chain or sending the synchronization receipt to the on-chain agent service successfully, the data synchronization program updates the "latest synchronized ID" to the last record ID of this processing, i.e. 2501, and persists it to Redis for next time polling.

[0064] It should be noted that when auditing a certain data synchronization operation, auditors can directly access the query node of the blockchain network module or through a trusted blockchain browser.

[0065] The auditors can query the synchronization receipt recorded in this time range according to the operation timestamp range (for example, 2023-10-27 10:00:00 to 10:01:00). The query result will clearly show the synchronization receipt content of the batch BATCH-20231027-005. The synchronization receipt proves that on 2023-10-27 10:00:05, the operation record with ID range 2501-2501 in the source database MySQL-DB01 was successfully synchronized to Elasticsearch-Cluster01 by SyncNode-A node. This record guaranteed by the blockchain module through the consensus mechanism has high public credibility and can be used as decisive evidence to solve data synchronization disputes.

[0066] Figure 3 A flowchart of a non-intrusive heterogeneous database real-time synchronization method based on blockchain according to an embodiment of the present application is shown, as shown in Figure 3 The method comprises the following steps: S1, using the data storage module configured in the source database, real-time automatic monitoring of the target data table's add, delete and modify operation, and capturing and storing the changed data. S2, using the data synchronization module configured outside the source database, obtaining the changed data to be synchronized from the data change storage module to push to the target database, and generating a synchronization receipt.

[0067] S3, receiving and constructing the synchronization reply generated by the data synchronization module by using the blockchain module, and performing transaction verification and recording the constructed synchronization reply in the blockchain module through a consensus mechanism.

[0068] In addition, the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the method in the above embodiment.

[0069] Based on the above embodiment, the present application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the steps of the method in the above embodiment.

[0070] In summary, the present application combines non-invasive data capture technology with blockchain storage technology to build a complete and reliable heterogeneous database synchronization and auditing solution. It not only realizes real-time data synchronization without invading the business system, meets the data flow demand under complex IT architecture, but more importantly, it uses blockchain to provide tamper-proof evidence for the synchronization process, greatly improving the transparency, security and auditability of data operations.

[0071] In addition, the present application has significant technical advantages and broad application prospects in fields such as financial transactions, judicial evidence, supply chain management, and government data sharing, which have strict requirements for data consistency and operation traceability.

[0072] Meanwhile, the non-invasive heterogeneous database real-time synchronization system based on blockchain provided by the present application also has the following advantages: (1) Realize non-invasive integration of business, reduce implementation cost and risk: The present application adopts a data capture technology based on database triggers (or other log analysis), without any code modification to the application program of the existing business system, which can realize real-time capture of data changes. This greatly reduces the complexity and risk of system integration, protects the existing IT investment of enterprises, and shortens the project implementation period.

[0073] (2) High heterogeneity compatibility and scalability: The data synchronization program of the present application adopts modular design, especially its data pushing unit can be flexibly adapted to different target databases, whether it is a relational database, a NoSQL database or a search engine. This makes the present application able to easily cope with the diverse heterogeneous data source synchronization needs in complex IT environments, with strong universality and scalability.

[0074] (3) Provide tamper-proof, high-trust synchronization process audit: the application generates a "synchronization receipt" by generating key information of each synchronization operation and notarizing it on the blockchain, and uses the decentralized, tamper-proof and traceable characteristics of the blockchain to provide "iron evidence" for the whole link of data synchronization. This on-chain notarization can be used as decisive evidence when solving data synchronization disputes, meeting compliance audits and conducting security accountability, and fundamentally ensures the transparency, security and accountability of the synchronization process.

[0075] (4) Efficient data integrity verification and audit capability: the application uses a Merkle tree to calculate the integrity hash of batch synchronization data. This method not only has high calculation efficiency and low resource consumption, but also supports "Merkle proof", so that the auditor can efficiently verify whether a specific record exists in the batch without obtaining all batch data, greatly improving the audit efficiency.

[0076] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0077] Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A blockchain-based non-intrusive real-time synchronization system for heterogeneous databases, characterized in that, The system includes: The data storage module, configured within the source database, is used to automatically monitor the addition, deletion, and modification operations of the target data table in real time, and to capture and store the changed data. The data synchronization module, configured outside the source database, is used to obtain the changed data to be synchronized from the data storage module, push it to the target database, and generate a synchronization receipt. The blockchain module is used to receive and construct the synchronization receipt generated by the data synchronization module, and to verify the constructed synchronization receipt through a consensus mechanism and record it in the blockchain module.

2. The system according to claim 1, characterized in that, The data storage module includes: The data change capture module is configured to be implemented through database triggers, and is used to automatically monitor the add, delete and modify operations of the target data table in real time and capture the changed data. A data change storage module is used to store the change data captured by the data change capture module.

3. The system according to claim 1, characterized in that, The data synchronization module includes a data synchronization table, which includes at least one of the following: table name, auto-incrementing ID, primary key value, operation type, data before change, data after change, and operation timestamp. The data synchronization table is used to store the changed data to be synchronized obtained from the data change storage module.

4. The system according to claim 3, characterized in that, The data synchronization module further includes a data synchronization program, which includes: A data extraction unit is used to manage synchronization points and extract change data to be synchronized in batches from the data synchronization table; The data push unit is used to parse the changed data to be synchronized, convert it into data operation instructions adapted to the target database format, and push the data operation instructions to the target database. The synchronization receipt generation unit is used to generate a structured synchronization receipt after the data operation instruction is pushed to the target database.

5. The system according to claim 4, characterized in that, The synchronization receipt includes at least one of the following: batch ID, ID range of processing records, source database identifier, target database identifier, synchronization timestamp, execution node information, and integrity hash value of batch data.

6. The system according to claim 5, characterized in that, The integrity hash value of the batch data is generated by constructing a Merkle tree, the specific process of which includes: Use a hash algorithm separately for each changed data to be synchronized in the same batch to generate leaf node hash values; The Merkle tree is constructed based on the hash values ​​of the leaf nodes to generate the Merkle root hash value as the integrity hash value of the batch data.

7. The system according to claim 1, characterized in that, The blockchain module includes: The blockchain service module is used to receive and construct the synchronization receipt generated by the data synchronization module, and to sign and broadcast the constructed synchronization receipt using a private key. The blockchain network module is used to receive the constructed synchronization receipt broadcast by the blockchain service module; and to verify the constructed synchronization receipt for transactions and record it in the blockchain network module through a consensus mechanism.

8. A non-intrusive real-time synchronization method for heterogeneous databases based on blockchain, characterized in that, The method includes the following steps: By utilizing the data storage module configured in the source database, the system can automatically monitor the addition, deletion, and modification operations of the target data table in real time, and capture and store the changed data. Using a data synchronization module configured outside the source database, the changed data to be synchronized is obtained from the data storage module and pushed to the target database, and a synchronization receipt is generated; The blockchain module receives and constructs the synchronization receipt generated by the data synchronization module, and verifies the constructed synchronization receipt through a consensus mechanism and records it in the blockchain module.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in claim 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in claim 8.

Images