Multi-cloud multi-file dynamic update integrity auditing method based on incremental Merkle tree

Abstract

The invention discloses a multi-cloud multi-file dynamic update integrity auditing method based on an incremental Merkle tree. Comprising the following steps: constructing file, cloud and global three-layer increment Merkle tree indexes to support real-time updating; periodically aggregating the global root Hash of a plurality of versions and constructing a state Merkle tree, and confirming the state root Hash of the state Merkle tree through a consensus mechanism participated by a cloud service provider to form a credible version anchor point; during auditing, a challenge is initiated by taking an anchor point as a reference, and a cloud service provider returns a four-level aggregation authentication path from a data block to a file root, a cloud root, a specific version global root and then to a state root; the auditor must first verify the version anchoring path and then verify the data path. Through combination of consensus anchoring and dependency verification, the problems of reference credibility, traceability and efficient verification of historical data state auditing in the multi-cloud environment are solved, and through optimization of dynamic priorities, state tree leaves with signatures, connection proof and the like, the safety, flexibility and performance of the system are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of cloud computing data security, and in particular to a method and system for auditing the data integrity of multiple frequently updated files in a storage environment of multiple cloud service providers. Background Technology

[0002] With the development of cloud computing, distributing data across multiple cloud service providers (multi-cloud) has become a common strategy to improve data reliability and avoid vendor lock-in. However, this model introduces new security challenges, namely, the need for users (or third-party auditors) to efficiently and reliably verify the integrity of data stored on remote, untrusted cloud servers.

[0003] In existing technologies, integrity verification schemes based on Merkle hash trees are widely used. The basic idea is to construct a hash tree for the data, with its root hash serving as a unique digest of data integrity. During auditing, the server provides the path (authentication path) from the challenged data block to the root hash, allowing the auditor to complete verification with a small number of hash operations. To support dynamic data updates, incremental update Merkle tree structures (such as authentication skip lists and dynamic 2-3 trees) have emerged, making update operations only require logarithmic-level computation.

[0004] However, applying such solutions directly to scenarios involving multiple clouds, multiple files, and frequent dynamic updates presents the following significant drawbacks:

[0005] Insufficient real-time performance: Existing integrity auditing methods based on Merkle trees require the entire Merkle tree to be reconstructed when a file is updated, resulting in high auditing latency and failing to meet real-time auditing requirements.

[0006] Cross-cloud consistency issues: In a multi-cloud environment, the file systems of different cloud service providers run independently, lacking a unified audit benchmark, making it difficult to achieve cross-cloud data consistency verification.

[0007] Low auditing efficiency: Traditional methods require a complete authentication path for each audit when dealing with frequently updated files, resulting in high computational overhead and impacting system performance.

[0008] Weak dynamic update processing capability: Existing methods have limited ability to process incremental updates of files and cannot efficiently reflect the latest data status.

[0009] Lack of security mechanisms: Existing methods lack effective anti-tampering mechanisms during the auditing process, making them vulnerable to man-in-the-middle attacks or data forgery.

[0010] Therefore, there is an urgent need for an integrity auditing solution that can provide efficient, reliable, and historical version traceability for multi-cloud dynamic data. Summary of the Invention

[0011] To address the aforementioned technical issues, this invention proposes a multi-cloud, multi-file dynamic update integrity auditing method and system based on incremental Merkle trees. Its core idea is to introduce a periodically generated state anchor point, agreed upon by the storage provider (cloud service provider), and logically bind all data integrity proofs to this anchor point, forming a verifiable chain of trust.

[0012] According to one aspect of the present invention, a method for dynamic update integrity auditing of multiple clouds and multiple files based on incremental Merkle trees is provided, comprising the following steps:

[0013] S1. Hierarchical Index Construction and Real-time Updates: Construct incremental update-enabled file Merkle subtrees for multiple files distributed across multiple cloud service providers; construct cloud Merkle subtrees for each cloud service provider, with leaf nodes representing the file root hash of the corresponding cloud file; aggregate the cloud root hashes of all cloud Merkle subtrees to generate a global root hash for the current version in real time; when a file undergoes a data block update, update the corresponding Merkle tree using incremental calculation to generate a new version of the global root hash.

[0014] S2. Consensus-based state anchor generation: The global root hashes of multiple consecutively generated versions are periodically aggregated into a batch to be confirmed; a state Merkle tree is constructed based on this batch to obtain the state root hash; the state root hash is submitted to a consensus mechanism jointly run by multiple cloud service providers participating in storage for confirmation; only after being confirmed by the consensus mechanism is the state root hash established as a version anchor for auditing with cross-cloud consistency.

[0015] S3. Perform an anchor-based audit challenge: The auditor initiates a challenge based on the confirmed state root hash, which includes a version identifier for locating the global root hash of a specific version; the audit proof returned by the cloud service provider includes an aggregated authentication path from the challenged data block to the file root hash, to the cloud root hash, to the global root hash of the specific version, and then to the state root hash.

[0016] S4. Verify the aggregate audit proof: The auditor verifies the hash consistency of the aggregate authentication path, which must first verify the path from the specific version global root hash to the state root hash to confirm that the global root hash has been legally anchored, and then verify the remaining paths; if all pass, the data integrity is confirmed.

[0017] Preferably, the periodic aggregation can be triggered according to a dynamic priority strategy: the system monitors the updated content of the global root hash; when an update involving a preset key file or a high-frequency audit file is detected, the construction of the state Merkle tree and consensus process is immediately triggered even if the preset time or quantity threshold is not reached.

[0018] Preferably, when constructing the state Merkle tree, its leaf node data not only includes the global root hash and its version identifier, but also includes the digital signature of each cloud service provider for its own cloud root hash under that version; the state root hash is calculated based on the leaf node data after the signature; in the audit proof returned in step S3, the fourth authentication path must be accompanied by the signature of the target cloud service provider; when verifying the fourth authentication path in step S4, the validity of the signature must be verified at the same time.

[0019] Preferably, the audit challenge further includes a random number generated by the auditor, which is mixed into the hash calculation of the first authentication path to prevent proof replay.

[0020] Preferably, the consensus mechanism employs an improved Practical Byzantine Fault Tolerance (PBFT) process, wherein the content of the pre-prepared message includes the root hash of the state Merkle tree and a list of hash values ​​of all its leaf node data, and each node must verify the correctness of the data related to itself in the list during the consensus process.

[0021] Preferably, when the global root hash corresponding to the challenged version identifier is located within the version range covered by two consecutive state anchors that have been agreed upon, but has not yet been included in the leaf nodes of any confirmed state Merkle tree, the audit proof returned by the cloud service provider should also include a connection proof: that is, the incremental update hash chain from the global root hash of the specific version to the global root hash of its previous anchored version, and the authentication path from the previous version to its state root hash.

[0022] Preferably, when verifying the aggregated authentication path, a hybrid strategy combining parallel verification and sequential dependency is adopted.

[0023] According to another aspect of the present invention, an auditing system for implementing the above-described method is provided.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] 1. Improved real-time performance: By updating the Merkle tree through incremental calculation, full reconstruction is avoided, significantly shortening the audit response time.

[0026] 2. Cross-cloud consistency: The state anchor points confirmed through the consensus mechanism ensure the consistency of audit benchmarks for cross-cloud data.

[0027] 3. High efficiency: The dynamic priority strategy optimizes the timing of state anchor point generation and reduces unnecessary consensus overhead.

[0028] 4. Enhanced security: The introduction of random number and digital signature mechanisms effectively prevents data tampering during the auditing process.

[0029] 5. Improved audit efficiency: The adoption of a hybrid verification strategy has optimized the verification process and improved audit efficiency. Attached Figure Description

[0030] Figure 1 This is the overall flowchart of the present invention;

[0031] Figure 2 This is a schematic diagram of the hierarchical index construction and real-time update system of the present invention;

[0032] Figure 3 This is a schematic diagram of the consensus state anchor point generation system of the present invention;

[0033] Figure 4 This is a schematic diagram of the anchor-based audit challenge execution system of the present invention;

[0034] Figure 5 This is a schematic diagram of the aggregated audit proof verification system of the present invention. Detailed Implementation

[0035] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0036] The following is in conjunction with the appendix Figure 1 -Appendix Figure 5 The specific embodiments of the present invention will be described in detail below.

[0037] Example 1: Hierarchical Index Construction and Real-time Update

[0038] See appendix Figure 2 This embodiment details the construction and incremental update process of file Merkle subtrees, cloud Merkle subtrees, and global root hashes, ensuring that the integrity of file data in a multi-cloud environment can be tracked efficiently and in real time.

[0039] 1.1 Construction of Merkle subtree in files

[0040] In a multi-cloud environment, each file is independently constructed as a Merkle subtree to record the integrity of its internal data blocks. The file Merkle subtree uses a binary tree structure, with leaf nodes representing the hash values ​​of the file's data blocks and non-leaf nodes representing combined hashes of the hash values ​​of their child nodes. For example, for a file containing 1024 data blocks, its Merkle tree will have 10 levels (from the leaf level to the root level), with the calculation process proceeding layer by layer from bottom to top.

[0041] When a file data block is updated, only the hash value of the leaf node corresponding to the affected data block needs to be updated, and the hash values ​​of all parent nodes on the path are updated upwards. This incremental update mechanism avoids the overhead of rebuilding the entire Merkle tree, greatly improving update efficiency. For example, when updating the 512th data block in the file, only 512 nodes (approximately 10 levels) on the path from that data block to the file root need to be updated, instead of recalculating the entire 1024 nodes of the Merkle tree.

[0042] 1.2 Cloud Merkle Subtree Construction

[0043] A cloud Merkle subtree is constructed for each cloud service provider, with its leaf nodes containing the root hash values ​​of the corresponding files on the cloud. For example, if cloud service provider A has 10 files, the leaf nodes of its cloud Merkle subtree contain the root hash values ​​of these 10 files, and the cloud root hash is obtained through layer-by-layer hash calculation.

[0044] The cloud Merkle subtree is constructed similarly to the file Merkle subtree, but its leaf nodes are file root hashes instead of original data blocks. When a file on the cloud service provider is updated, only the root hash value of that file needs to be updated, and then the cloud Merkle subtree is updated; there is no need to recalculate the Merkle tree for all files.

[0045] 1.3 Global Root Hash and Incremental Update

[0046] The global root hash is generated by aggregating the root hash values ​​of all cloud Merkle subtrees. For example, if there are 5 cloud service providers, the global root hash is a combined hash of these 5 cloud root hashes.

[0047] When a file's data block is updated, the system updates the Merkle tree using an incremental calculation method:

[0048] 1. Only update the Merkle subtree of the affected files.

[0049] 2. Only update the affected cloud Merkle subtrees.

[0050] 3. Only update the global Merkle tree.

[0051] This hierarchical incremental update mechanism ensures that the system can respond efficiently to file updates, avoiding the high overhead of full reconstruction. For example, when one of the 1024 data blocks in a file is updated, only about 10 nodes of the file's Merkle subtree, about 5 nodes of the corresponding cloud Merkle subtree, and about 3 nodes of the global Merkle tree need to be updated, rather than recalculating the Merkle trees of all files, all cloud service providers, and the global tree.

[0052] Example 2: Consensus-Based State Anchor Generation

[0053] See appendix Figure 3 This embodiment describes in detail the generation mechanism of state anchor points to ensure cross-cloud consistency of data audit benchmarks in a multi-cloud environment.

[0054] 2.1 Construction of the Merkle Tree of State

[0055] The system periodically aggregates multiple continuously generated versions of the global root hash into a batch awaiting confirmation, constructing a state Merkle tree. The leaf nodes of the state Merkle tree contain the global root hash, the version identifier, and the cloud service provider's digital signature of the cloud root hash.

[0056] For example, the system generates a state anchor every 10 minutes or every 100 versions. When the system detects an update to a critical file (such as financial data) or a frequently audited file (such as user login records), it immediately triggers the construction and consensus process of the state Merkle tree, even if the time or quantity thresholds have not been reached.

[0057] 2.2 Dynamic Priority Strategy

[0058] The system monitors critical files and frequently audited files, employing a dynamic priority strategy to trigger the generation of status anchors. The set of critical files includes documents such as " " Sensitive documents such as "" and frequently audited documents include such documents. " Documents that are frequently audited include…

[0059] When updates to these files are detected, the system immediately generates state anchor points to ensure that audit baselines for critical data are updated in a timely manner, improving the real-time nature of auditing. For example, when " "When a file is updated, the system will immediately trigger the generation of state anchors, even if the current batch contains only 50 versions (not reaching the threshold of 100 versions)."

[0060] 2.3 Improved Consensus Mechanism

[0061] The state root hash is submitted to a consensus mechanism jointly operated by multiple cloud service providers for confirmation. The consensus mechanism employs an improved practical Byzantine fault-tolerant protocol to ensure the security and reliability of the state root hash confirmation process.

[0062] During the consensus process, the pre-prepared message contains the root hash of the state Merkle tree and a list of hash values ​​for all its leaf nodes. During the preparation and confirmation phases, each node, in addition to verifying the message format, must verify the correctness of its own maintained global root hash corresponding to the relevant leaf nodes in the list. This design ensures the security and consistency of the consensus process and effectively prevents interference from malicious nodes.

[0063] Example 3: Auditing Challenges and Verification

[0064] See appendix Figure 4 This embodiment details the process of initiating and verifying audit challenges to ensure the security and efficiency of data integrity audits.

[0065] 3.1 Audit Challenges and Evidence Generation

[0066] The auditor initiates a challenge based on the confirmed state root hash, which includes a version identifier used to locate the global root hash of a specific version. The audit proof returned by the cloud service provider contains an aggregated authentication path from the challenged data block to the file root hash, to the cloud root hash, to the global root hash of the specific version, and then to the state root hash.

[0067] To enhance audit security, the auditor generates a random number. When generating audit proof, the cloud service provider uses this random number as additional input to the hash calculation, mixing it into the hash calculation process of the first authentication path from the challenged data block to the file root hash. For example, the original hash calculation is... After mixing in random numbers, it becomes ,in It is a random number.

[0068] 3.2 Audit Verification Process

[0069] The auditor verifies the hash consistency of the aggregated authentication path, first verifying the path from the specific version's global root hash to the state root hash to confirm that the global root hash has been legitimately anchored. After successful verification, the remaining paths are then verified.

[0070] The mathematical expression of the verification process is as follows:

[0071] 1. Verification :

[0072] Set the global hash value ( ) and path Perform XOR operation ( Then, calculate the hash of the result. If the result equals the pre-stored state hash ( Then prove The data has not been tampered with.

[0073] 2. Verification :

[0074] Same use and After XORing, the hash is calculated. If the result is the same as the pre-stored cloud hash ( If they are consistent, then The data is valid.

[0075] 3. Verification :

[0076] Use the cloud hash verified in the previous step ( )and After XORing, calculate the hash. If the result equals the pre-stored file hash ( ), then prove the file data (through The transmission was not tampered with.

[0077] 4. Verification :

[0078] Left side: Encrypted data block With a random value (or key) Calculate the hash after XORing;

[0079] Right side: File hash ( )and( The hash is calculated after XORing;

[0080] If the results on both sides are equal, then it proves... The data is consistent with the file hash and encrypted data blocks, ensuring the integrity of the data from storage to transmission.

[0081] 3.3 Hybrid Validation Strategy

[0082] During the verification process, a hybrid strategy combining parallel verification and sequential dependency is employed: while initiating parallel hash calculations for the first, second, and third authentication paths, the verification result of the fourth authentication path is executed and determined first. The entire verification is considered successful only when the fourth path verification passes and the result of the parallel calculation matches the hash value of the corresponding node.

[0083] This strategy ensures both the accuracy and efficiency of verification, making it particularly suitable for handling a large number of concurrent audit requests in a multi-cloud environment.

[0084] Example 4: Connection Proof Mechanism

[0085] See appendix Figure 5 This embodiment describes in detail the connection proof mechanism to ensure that the integrity of any version within the version range can be effectively verified.

[0086] When the global root hash corresponding to the challenge's version identifier is within the version range covered by two consecutive state anchors that have been agreed upon, but has not yet been included in the leaf nodes of any confirmed state Merkle tree, the audit proof returned by the cloud service provider must include the connection proof.

[0087] The proof of connection includes:

[0088] 1. An incremental update hash chain from a specific version of the global root hash to the global root hash of its previous anchored version.

[0089] 2. The authentication path from the previous version to its state root hash.

[0090] For example, suppose state anchor A corresponds to version 100, state anchor B corresponds to version 200, and the audit request is for version 150, which is not included in any confirmed state Merkle tree. Then the connection proof includes the incremental update hash chain from version 150 to 100, and the authentication path from version 100 to state anchor A.

[0091] The conditions for the auditor to verify the connection proof are as follows:

[0092]

[0093] in The root hash of state anchor point A, For the global root hash of version 100, For the authentication path from version 100 to state anchor A, For the global root hash of version 150, Update the hash chain incrementally from version 150 to 100.

[0094] Example 5: Digital Signature Mechanism

[0095] See appendix Figure 5 This embodiment describes the digital signature mechanism in detail to enhance the credibility of audit proof.

[0096] During the construction of the state Merkle tree, the leaf node data includes not only the global root hash and its version identifier, but also the digital signature of each cloud service provider's own cloud root hash. For example, cloud service provider A's own cloud root hash... signature Generated by signing with a private key, in the following format: .

[0097] In the audit verification document, the fourth authentication path must include the target cloud service provider's signature of its own cloud root hash in the leaf node corresponding to the specific version of the global root hash. During verification, the auditor must also verify the validity of this signature to ensure that the cloud service provider cannot forge the audit verification document.

[0098] Through the above embodiments, this invention achieves real-time performance, cross-cloud consistency, and high efficiency in file data integrity auditing under multi-cloud environments, solving the problems of real-time performance, cross-cloud consistency, and auditing efficiency in traditional methods. This embodiment describes the implementation details of the technical solution in detail, ensuring the implementability and technical effectiveness of this invention. Those skilled in the art can make appropriate adjustments and improvements to this invention according to actual needs, and such adjustments and improvements all fall within the protection scope of this invention.

Claims

1. A method for dynamic update integrity auditing of multiple clouds and multiple files based on incremental Merkle trees, characterized in that, Includes the following steps: S1. Hierarchical Index Construction and Real-time Update: Construct file Merkle subtrees for multiple files distributed across multiple cloud service providers; construct cloud Merkle subtrees for each cloud service provider, with leaf nodes representing the file root hash of the corresponding cloud file; aggregate the cloud root hashes of all cloud Merkle subtrees to generate a global root hash for the current version in real time; when a file undergoes data block updates, update the corresponding Merkle tree using incremental calculation to generate a new version of the global root hash. S2. Consensus-based state anchor generation: The global root hashes of multiple consecutively generated versions are periodically aggregated into a batch to be confirmed; a state Merkle tree is constructed based on this batch to obtain the state root hash; the state root hash is submitted to a consensus mechanism jointly run by multiple cloud service providers participating in storage for confirmation; only after obtaining confirmation from the consensus mechanism is the state root hash established as a version anchor with cross-cloud consistency for auditing. S3. Perform an anchor-based audit challenge: The auditor initiates a challenge based on the confirmed state root hash, which includes a version identifier for locating the global root hash of a specific version; the audit proof returned by the cloud service provider includes an aggregated authentication path from the challenged data block to the file root hash, to the cloud root hash, to the global root hash of the specific version, and then to the state root hash. S4. Verify the aggregate audit proof: The auditor verifies the hash consistency of the aggregate authentication path, which must first verify the path from the specific version global root hash to the state root hash to confirm that the global root hash has been legally anchored, and then verify the remaining paths; if all pass, the data integrity is confirmed.

2. The method according to claim 1, characterized in that, In step S2, the periodic aggregation is triggered based on a dynamic priority strategy: the system monitors the updated content of the global root hash; when an update involving a preset key file or a high-frequency audit file is detected, the construction of the state Merkle tree and consensus process is immediately triggered even if the preset time or quantity threshold is not reached.

3. The method according to claim 2, characterized in that, In step S2, when constructing the state Merkle tree, its leaf node data not only includes the global root hash and its version identifier, but also includes the digital signature of each cloud service provider for its own cloud root hash under that version; the state root hash is calculated based on the leaf node data after the signature. In the audit proof returned in step S3, the fourth authentication path must include the signature of the target cloud service provider on its own cloud root hash in the leaf node corresponding to the specific version of the global root hash; when verifying the fourth authentication path in step S4, the validity of the signature must also be verified.

4. The method according to claim 1, characterized in that, In step S3, the audit challenge also includes a random number generated by the auditor; when generating the aggregated authentication path, the cloud service provider needs to use this random number as an additional input for the hash calculation and mix it into the hash calculation process of the first authentication path from the challenged data block to the file root hash; the verification process in step S4 needs to be reviewed using the same random number.

5. The method according to claim 1, characterized in that, The consensus mechanism in step S2 adopts an improved practical Byzantine fault-tolerant process, wherein the content of the pre-preparation message includes the root hash of the state Merkle tree and a list of hash values ​​of all its leaf node data; in the preparation and confirmation phases, in addition to verifying the message format, each node must verify the correctness of the global root hash it maintains, which corresponds to the relevant leaf node in the list.

6. The method according to claim 1, characterized in that, In step S3, when the global root hash corresponding to the challenged version identifier is within the version range covered by two consecutive state anchors that have been agreed upon, but has not yet been included in the leaf node of any confirmed state Merkle tree, the audit proof returned by the cloud service provider must also include a connection proof: that is, the incremental update hash chain from the global root hash of the specific version to the global root hash of its previous anchored version, and the authentication path from the previous version to its state root hash.

7. The method according to claim 1, characterized in that, In step S4, when verifying the aggregated authentication path, a hybrid strategy combining parallel verification and sequential dependency is adopted: while starting the parallel hash calculation for the first, second, and third authentication paths, the verification result of the fourth authentication path is executed and determined first. The entire verification is considered successful only if the fourth path verification passes and the result of the parallel computation matches the hash value of the corresponding node.

8. An auditing system for implementing the method of any one of claims 1-7, characterized in that, include: The user-side component is used to initiate data operation requests. The distributed consensus storage module contains multiple cloud service provider nodes. Each node stores data and maintains a corresponding Merkle subtree. All nodes together form a consensus network for confirming state anchors. The state anchor management module is used to aggregate multiple versions of the global root hash, construct a state Merkle tree, and call the consensus network to confirm the state root hash; wherein, the state anchor management module or any cloud service provider node is configured to execute the dynamic priority monitoring strategy described in claim 2; An auditing module with a hybrid validator is used to obtain the confirmed state root hash, initiate a challenge containing a version identifier and a random number, and verify the returned aggregate authentication path and additional signature or connection proof according to the hybrid strategy of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images