Data auditing method and device based on merkle tree and blockchain

Abstract

The present disclosure relates to a data auditing method and device based on a Merkle tree and a blockchain. The method comprises: sending a data auditing request including first data to a second server, so that the second server determines a global index of the first data in the total data of the second server and a Merkle proof of the global index based on the data auditing request; receiving the global index and the Merkle proof sent by the second server, calculating a first hash value of the first data in the same hash calculation manner as a leaf hash value in the Merkle tree, and calculating a first root hash value based on the first hash value and the Merkle proof; obtaining a second root hash value corresponding to the global index from the blockchain, and comparing the first root hash value with the second root hash value. If the first root hash value is consistent with the second root hash value, it is determined that the data auditing is passed. The present scheme realizes the verification of the integrity and authenticity of the data while protecting the data privacy.

Description

Data Auditing Method and Device Based on Merkle Tree and Blockchain Technical Field

[0001] This disclosure relates to the fields of data security and computer technology, and in particular to a data auditing method and apparatus based on Merkle trees and blockchain. Background Technology

[0002] In related technologies, in cross-institutional data collaboration scenarios, when service fees may be inversely proportional to the total number of users maintained by the data provider, data users often cannot verify the provider's claim of full data volume due to privacy restrictions, leading to a lack of trust. Traditional solutions have significant drawbacks: introducing third-party auditing institutions is costly and may create centralized risks; requiring full data exposure seriously infringes on commercial privacy; and simple overall hash verification can only verify whether the data has been tampered with, and cannot effectively prevent providers from exaggerating their scale by maliciously omitting some data. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a data auditing method and apparatus based on Merkle trees and blockchain.

[0004] According to a first aspect of the present disclosure, a data auditing method based on Merkle trees and blockchain is provided, applied to a first server, comprising:

[0005] A data audit request, including the first data, is sent to a second server, so that the second server, based on the data audit request, determines the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is the data provided by the second server to the first server; the Merkle tree is generated based on the full data and the global index of the full data; the global index of the full data is continuous and unique;

[0006] Receive the global index and the Merkel proof sent by the second server, calculate the first hash value of the first data based on the global index using the same hash calculation method as the leaf hash value in the Merkel tree, and calculate the first root hash value based on the first hash value and the Merkel proof.

[0007] The second root hash value corresponding to the global index is obtained from the blockchain. The first root hash value and the second root hash value are compared. If the first root hash value and the second root hash value are consistent, the data audit is confirmed to be successful. The second root hash value is the root hash value of the Merkle tree that is pre-stored by the second server.

[0008] According to a second aspect of the present disclosure, a data auditing method based on Merkle trees and blockchain is provided, applied to a second server, comprising:

[0009] In response to receiving a data audit request including first data sent by a first server, based on the data audit request, the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree are determined; the first data is data provided by the second server to the first server; the Merkle tree is generated based on the full data and the global index of the full data; the global index of the full data is continuous and unique;

[0010] The global index and the Merkel proof are sent to the first server, so that the first server calculates the first hash value of the first data based on the global index, using the same calculation method as the leaf hash values ​​in the Merkel tree. Based on the first hash value and the Merkel proof, the first root hash value is calculated. The second root hash value corresponding to the global index is obtained from the blockchain. The first root hash value and the second hash value are compared. If the first root hash value and the second hash value are consistent, the data audit is determined to be successful. The second hash value is the root hash value of the Merkel tree pre-stored by the second server.

[0011] According to a third aspect of the present disclosure, a data auditing device based on Merkle trees and blockchain is provided, applied to a first server, comprising:

[0012] A sending unit is configured to send a data audit request, including first data, to a second server, so that the second server, based on the data audit request, determines the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is data provided by the second server to the first server; the Merkle tree is generated based on the full data and the respective global indexes of the full data; the global indexes of the full data are continuous and unique;

[0013] The calculation unit is configured to receive the global index and the Merkel proof sent by the second server, calculate the first hash value of the first data based on the global index using the same hash calculation method as the leaf hash values ​​in the Merkel tree, and calculate the first root hash value based on the first hash value and the Merkel proof.

[0014] The comparison unit is used to obtain the second root hash value corresponding to the global index from the blockchain, compare the first root hash value with the second hash value, and if the first root hash value and the second hash value are consistent, it is determined that the data audit has passed; the second hash value is the root hash value of the Merkle tree pre-stored by the second server.

[0015] According to a fourth aspect of the present disclosure, a data auditing device based on Merkle trees and blockchain is provided, applied to a second server, comprising:

[0016] A determining unit is configured to, in response to receiving a data audit request including first data sent by a first server, determine, based on the data audit request, the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is data provided by the second server to the first server; the Merkle tree is generated based on the full data and its respective global index; the global index of the full data is continuous and unique;

[0017] The sending unit is configured to send the global index and the Merkel proof to the first server, so that the first server calculates the first hash value of the first data based on the global index, using the same calculation method as the leaf hash values ​​in the Merkel tree; calculates the first root hash value based on the first hash value and the Merkel proof; and retrieves the second root hash value corresponding to the global index from the blockchain, compares the first root hash value with the second hash value, and if the first root hash value and the second hash value are consistent, it is determined that the data audit has passed; the second hash value is the root hash value of the Merkel tree pre-stored by the second server.

[0018] According to a fifth aspect of the present disclosure, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of the first or second aspects.

[0019] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0020] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0021] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: by introducing a continuous and unique global index and incorporating it into hash calculation, the verification of a single piece of data can be precisely bound to the full data scale, effectively eliminating the possibility of data providers cheating by hiding part of the data. The auditor only needs to complete the verification independently based on its own existing first data and the index and proof returned by the data provider, without having to provide the full data to the auditor. Thus, the audit can be completed under the premise of ensuring data sovereignty and privacy. Furthermore, by using the blockchain to immutably store the Merkle root hash value, a reliable comparison benchmark and evidence are provided for the entire audit process, making the audit conclusions publicly verifiable, thereby establishing a solid trust mechanism in a decentralized environment.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] Figure 1 is a flowchart illustrating a data auditing method based on Merkle trees and blockchain according to an exemplary embodiment.

[0025] Figure 2 is a schematic diagram of a data auditing system based on Merkle tree and blockchain according to an exemplary embodiment.

[0026] Figure 3 is a flowchart illustrating another data auditing method based on Merkle trees and blockchain according to an exemplary embodiment.

[0027] Figure 4 is a schematic diagram of a Merkle tree proposed in an embodiment of this disclosure.

[0028] Figure 5 is a block diagram illustrating a data auditing device based on Merkle tree and blockchain according to an exemplary embodiment.

[0029] Figure 6 is a block diagram illustrating another data auditing device based on Merkle trees and blockchain according to an exemplary embodiment.

[0030] Figure 7 is a block diagram of an apparatus for a data auditing method based on Merkle trees and blockchain, according to an exemplary embodiment.

[0031] Figure Labels

[0032] 501-Transmitting unit; 502-Calculation unit; 503-Comparison unit; 601-Determining unit; 602-Transmitting unit; 700-Device; 702-Processing component; 704-Memory; 706-Power component; 708-Multimedia component; 710-Audio component; 712-I / O interface; 716-Communication component; 720-Processor. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0034] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0035] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the words “if” and “suppose” as used herein may be interpreted as “when”, “when”, or “in response to a determination”.

[0036] Furthermore, various forms of processes shown in the embodiments of this disclosure can be used to reorder, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0037] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In today's increasingly collaborative digital business environment, data sharing and collaborative computing among enterprises are becoming more and more common. A typical scenario exists where the data provider (e.g., Company B) possesses massive amounts of core data, while the data user (e.g., Company A) only needs to use or verify a small portion of it. When the commercial settlement between the two parties is tied to the full scale of the data provider's data, a trust issue arises.

[0039] For example, in cloud computing or API service payment scenarios, service fees may be inversely proportional to the total number of users D maintained by data provider B. Data user A only possesses a subset d of users of its own service and cannot directly access or verify B's full dataset D, primarily due to considerations of data sovereignty, user privacy, and trade secret protection. This information asymmetry creates an incentive for data provider B to falsify data, such as underreporting its actual total data volume D to obtain unfairly higher fees.

[0040] Existing solutions have obvious limitations:

[0041] Traditional third-party auditing involves bringing in a trusted third-party organization to conduct the audit, but this adds extra costs, and the centralized organization itself may become a performance bottleneck or a point of trust risk.

[0042] Full data comparison: Requiring B to provide full data to A or the auditor is a serious violation of data privacy protection principles and is not feasible in most business scenarios;

[0043] Simple hash verification: Performing a hash calculation on the entire dataset at once can verify whether the data has been tampered with, but it cannot prove the integrity of the data, that is, it cannot prevent the provider from maliciously omitting some data to reduce the size of the dataset.

[0044] Therefore, there is an urgent need for a technical solution that allows auditors to efficiently and reliably verify the authenticity of the total amount of data claimed by the data provider and the integrity of the data itself without exposing the full content of the data, thereby building a fair and transparent decentralized trust mechanism.

[0045] To address the aforementioned issues, this disclosure provides a data auditing method and apparatus based on Merkle trees and blockchain. By introducing a continuous and unique global index and incorporating it into hash calculations, the verification of a single data entry can be precisely bound to the entire data volume. This effectively eliminates the possibility of data providers cheating by hiding partial data. The auditor only needs to use its own existing initial data and the index and proof returned by the data provider to independently complete the verification without providing the full data to the auditor. This allows the audit to be completed while ensuring data sovereignty and privacy. Furthermore, by using blockchain to immutably store the Merkle tree root hash value, a reliable comparison benchmark and evidence are provided for the entire audit process, making the audit conclusions publicly verifiable. This establishes a robust trust mechanism in a decentralized environment.

[0046] Figure 1 is a flowchart illustrating a data auditing method based on Merkle trees and blockchain according to an exemplary embodiment. As shown in Figure 1, it should be noted that the data auditing method based on Merkle trees and blockchain in this embodiment is applied to a first server. As shown in Figure 1, the method may include the following steps:

[0047] Step 101: Send a data audit request, including the first data, to the second server so that the second server can determine the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree, based on the data audit request.

[0048] The first data is the data provided by the second server to the first server.

[0049] The Merkle tree is generated based on the full dataset and its respective global index.

[0050] Among them, the global index of the full data is continuous and unique.

[0051] In one embodiment, the first server may be a server used by the data auditor, and the second server may be a server used by the data provider.

[0052] In some embodiments, the auditor (first server) issues a challenge (i.e., a data audit request) to the data provider (second server), the core of which is to request the data provider to prove that the first data held by the auditor is fully contained in the full dataset claimed by the data provider. In response to this challenge, the data provider determines the global sequence number (i.e., global index) corresponding to the first data in its full dataset according to rules pre-established with the auditor (e.g., sorted by user ID). Furthermore, since the full dataset has been constructed into one or more Merkle trees based on this global index system (where the hash of each tree's leaf node is calculated jointly by the data content and its global index), the data provider must locate the specific Merkle tree containing the data based on this global index and generate a path proof (i.e., a Merkle proof) from the leaf node corresponding to the data to the root of the tree. This process ensures that the generation of the proof strictly depends on the data's definitive location and content within the global dataset.

[0053] It's important to note that a Merkel proof is essentially a path composed of hash values. When proving that specific data (such as a transaction or user record) is contained in a Merkel tree, only the hash values ​​of all required sibling nodes along the path from the leaf node corresponding to that data to the root are needed, without exposing the entire tree or any other data. The auditor uses the leaf hash calculated from the specific data (the first data) itself, combined with the sibling hashes provided in the proof, to recalculate the hash layer by layer from the bottom up according to the tree structure, ultimately obtaining a root hash. If this calculated root hash is completely consistent with a pre-published and tamper-proof root hash (such as one stored on a blockchain), it proves that the data exists in the original and complete dataset.

[0054] In this embodiment of the disclosure, the continuous and unique global index of the full data is the technical cornerstone that can prevent data providers from cheating by hiding or omitting data. It can force each record in the full dataset to be assigned a sequence number that starts from 0, is uninterrupted and non-repeating, thereby binding the data itself to its precise position in the global sequence.

[0055] Understandably, the total number of global indexes directly represents the total amount of data N. The absence of any record (e.g., being intentionally hidden) will disrupt the continuity of the index. In the subsequent random challenge on the first server, if the auditor selects data that was not reported, the data provider cannot forge a valid proof that simultaneously satisfies the requirements of correct content, continuous index, and verification through a Merkle tree. All indexes after the hidden data will be misaligned and will fail the audit.

[0056] In some embodiments of this disclosure, step 101 may specifically include the following steps: randomly selecting one or more pieces of data as first data from all the data historically provided by the second server, generating a data audit request including the first data, and sending the data audit request to the second server.

[0057] In this embodiment, the auditor (first server) can randomly select one or more data records from its locally stored collection of all historical data previously delivered by the data provider (second server) as the challenge sample (i.e., the first data) for this audit. Subsequently, the auditor encapsulates the selected data into a structured data audit request and sends it to the data provider. This random selection ensures the unpredictability of the audit challenge, preventing the data provider from knowing in advance or deliberately falsifying evidence of specific data. By executing such random challenges multiple times, the auditor can infer the integrity and authenticity of the full dataset claimed by the data provider with extremely high statistical confidence, thereby effectively deterring and detecting data misrepresentation by the data provider.

[0058] Step 102: Receive the global index and Merkel proof sent by the second server. Based on the global index, calculate the first hash value of the first data in the same hash calculation method as the leaf hash value in the Merkel tree. Based on the first hash value and the Merkel proof, calculate the first root hash value.

[0059] In this embodiment of the disclosure, after the auditing party (first server) receives the global index and Merkel proof returned by the data provider (second server), it can combine the first data it holds with the global index it received according to the same hash calculation rule strictly agreed upon by both parties in advance to generate a local leaf hash value (i.e., the first hash value), thereby ensuring that the starting point of verification is completely consistent with the starting point when the data provider constructs the Merkel tree.

[0060] As an example, the auditor can use the user ID, key fields, and global index provided by the data provider for the record to independently calculate the hash value of the leaf node according to the hash calculation rules mentioned above.

[0061] In one embodiment, the auditor uses the first hash value and a Merkle proof (which is essentially a series of sibling node hash values ​​on the hash path) to progressively calculate the hashes of upper-level nodes using a fixed bottom-up algorithm for the Merkle tree, ultimately obtaining a complete root hash value (i.e., the first root hash value). Through this process, the auditor can simulate and reconstruct the final root hash corresponding to the data in the Merkle tree using only a single data point, an index, and a short set of path proofs, providing a reliable computational foundation for the next step of comparison with the immutable on-chain evidence.

[0062] Step 103: Obtain the second hash value corresponding to the global index from the blockchain, compare the first hash value with the second hash value, and if the first hash value and the second hash value are consistent, the data audit is confirmed to be successful.

[0063] The second root hash value is the root hash value of the Merkle tree that is pre-stored on the second server.

[0064] In some embodiments, the total amount of data can also be obtained from the blockchain.

[0065] In one embodiment, the auditor can use the received global index, combined with pre-agreed grouping rules (e.g., calculating group number = floor(global index / fixed group size)), to precisely locate the root hash value of the Merkle tree corresponding to the data in the blockchain's notarization record. This is the immutable second root hash value, representing the data provider's public commitment to the integrity of the specific group of data at the time of notarization. Subsequently, the auditor compares the first root hash value calculated in step 102 with the second root hash value obtained from the chain. If they are completely identical, it proves that the challenged first data content and the global index are both true and accurate, and are indeed included in the Merkle tree initially constructed and anchored to the blockchain by the data provider, thus passing the data audit. Conversely, any inconsistency directly reveals deficiencies in the data provider's data integrity, index authenticity, or proof validity, resulting in audit failure. Thus, leveraging the immutability of the blockchain, the complex integrity audit problem is transformed into a simple, deterministic, and publicly verifiable hash value comparison.

[0066] It is important to emphasize that this technical solution fundamentally eliminates the incentive for data providers to falsify data by constructing a dual guarantee system that reinforces both technology and economics. Technically, it relies on a "random challenge-response" auditing mechanism based on a global index and Merkle trees. Data providers must pre-construct a Merkle tree from their dataset containing a continuous and unique global index, anchoring the root hash and the total data volume N to the blockchain. During auditing, the auditor randomly selects samples from their known data to initiate challenges. Because the global index system strictly binds each data entry to a fixed sequence number, any underreporting of the total data volume will cause the index of some genuine data to exceed the promised range. Under random auditing with unpredictable challenge samples, systemic fraud is technically extremely difficult to conceal and easily exposed. Economically, the aforementioned verifiable and undeniable technical mechanism provides an enforceable basis for stringent punitive clauses in commercial contracts. Once a technical audit discovers an inconsistency, the high compensation clause in the contract can be automatically triggered, making the expected cost of fraud far exceed its potential benefits, thus ensuring that fraud can be detected. The economic contract ensures that if it is discovered, it will be severely punished. The two form a closed loop that creates a solid barrier to eliminate the impulse to commit fraud.

[0067] In some embodiments of this disclosure, obtaining the second root hash value corresponding to the global index from the blockchain in step 103 may specifically include:

[0068] Based on the preset grouping method of the full data, the target group to which the global index belongs is determined; the full data corresponds to multiple Merkle trees, and each Merkle tree is generated based on the data in a group; the grouping is obtained by the second server dividing the full data according to the preset grouping method;

[0069] Obtain the root hash value of the Merkle tree corresponding to the target group from the blockchain to obtain the second root hash value.

[0070] In this embodiment, due to the massive scale of the full data, the data provider (second server) has divided it into several groups during the preprocessing stage according to rules (such as a fixed number of records) jointly determined by both parties, and independently constructed a Merkle tree for each group. Therefore, when the auditor (first server) obtains the global index of a certain data, it can uniquely determine the specific target group to which the data belongs based on the same preset grouping method (e.g., performing a division operation of target group number = global index / fixed-size integer for each group). The auditor can then directly query the blockchain for the root hash value of the Merkle tree that is pre-stored by the data provider and strictly corresponds to the calculated target group number, thereby obtaining the second root hash value as the final comparison benchmark. This transforms the audit verification of massive amounts of data into the location and verification of small, fixed group units, greatly improving the system's scalability and verification efficiency.

[0071] In some embodiments, as shown in Figure 2, a data auditing system based on Merkle trees and blockchain is provided, including auditor A (first server), data provider B (second server), and blockchain. Auditor A initiates a challenge to data provider B (i.e., step 1, initiating the challenge), submitting the identifier of the data to be verified (such as a user ID); B locates the data, generates the corresponding global index and its path proof in the Merkle tree (i.e., step 2, Merkle proof search record, generating Merkle proof), and returns it to A (i.e., step 3, returning); A uses this proof and its own calculated leaf hash to deduce the root hash to be verified (i.e., step 4, calculating the leaf hash and using the proof to deduce the root hash); simultaneously, A obtains from the blockchain the root hash list previously stored by B and the total number of data N of the full data (i.e., steps 5-6, querying and returning the stored data of version t1, including the total number N and the root hash list); finally, A locates the corresponding root hash value on the chain through the global index, compares it with its own deduced root hash, if they match, the verification passes, otherwise the verification fails, thus completing a traceable and tamper-proof audit of the existence and integrity of the data (i.e., step 7, locating the root hash on the chain according to the global index and comparing it with the deduced root hash).

[0072] According to the data auditing method based on Merkle tree and blockchain proposed in this disclosure, by introducing a continuous and unique global index and incorporating it into hash calculation, the verification of a single piece of data can be precisely bound to the full data scale, effectively preventing the possibility of data providers cheating by hiding part of the data. The auditor only needs to complete the verification independently based on its own existing first data and the index and proof returned by the data provider, without having to provide the full data to the auditor. Thus, the audit can be completed under the premise of ensuring data sovereignty and privacy. Furthermore, by using blockchain to immutably store the Merkle tree root hash value, a reliable comparison benchmark and evidence are provided for the entire audit process, making the audit conclusion publicly verifiable, thereby establishing a solid trust mechanism in a decentralized environment.

[0073] Figure 3 is a flowchart illustrating a data auditing method based on Merkle trees and blockchain according to an exemplary embodiment. As shown in Figure 3, it should be noted that the data auditing method based on Merkle trees and blockchain in this embodiment is applied to a second server. As shown in Figure 3, the method may include the following steps:

[0074] Step 301: In response to receiving a data audit request including the first data sent by the first server, based on the data audit request, determine the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree.

[0075] The first data is the data provided by the second server to the first server.

[0076] The Merkle tree is generated based on the full dataset and its respective global index.

[0077] Among them, the global index of the full data is continuous and unique.

[0078] Step 302: Send the global index and Merkel proof to the first server so that the first server can calculate the first hash value of the first data based on the global index, using the same calculation method as the leaf hash value in the Merkel tree. Based on the first hash value and the Merkel proof, calculate the first root hash value. Also, obtain the second root hash value corresponding to the global index from the blockchain. Compare the first root hash value with the second hash value. If the first root hash value and the second hash value are consistent, the data audit is confirmed to be successful.

[0079] The second root hash value is the root hash value of the Merkle tree that is pre-stored on the second server.

[0080] In the embodiments of this disclosure, steps 301-302 can be implemented using the data auditing method based on Merkle tree and blockchain proposed in any embodiment of this disclosure, which will not be elaborated here.

[0081] In some embodiments of this disclosure, the method may further include the following steps:

[0082] Perform a global sort on each data point in the full dataset according to a preset sorting method;

[0083] Each data point after global sorting is assigned a separate global index; the global indices of the data in the entire dataset are consecutive and unique.

[0084] The full dataset is grouped according to a preset data volume to obtain multiple groups;

[0085] Based on the global index of each data point, a Merkle tree is generated for each group, and the second root hash value of each Merkle tree is obtained;

[0086] Store the second root hash value of all groups into the blockchain.

[0087] In one embodiment, all records in the full dataset D can be globally sorted according to a deterministic rule (such as sorting by user ID in ascending order of the primary key field). After sorting, a contiguous and unique global index (global_index) is assigned to each record, starting from 0. At this point, the total number of records N in the dataset is obtained.

[0088] As an example, the sorted full dataset D can be logically grouped according to a preset fixed size n (n is a power of 2, such as 1024). A total of m = ceil(N / n) data groups are generated.

[0089] In one embodiment, a Merkle tree is constructed for each group. This involves taking the key field (such as user ID or key business field) of each record within the group, concatenating it with the record's global index, and then calculating the hash value of the leaf node of the Merkle tree using a hash function (such as SHA-256). The calculation rule is: leaf_hash = Hash(user ID + key field + global_index). By calculating the hash value layer by layer upwards, a unique Merkle root hash Merkle_Root_i is finally generated for each data group (where i is the group number, from 0 to m-1).

[0090] In some embodiments of this disclosure, for the last group, if the number of records is less than n, predefined placeholder records (e.g., records with empty content or specific values) can be used to fill the remaining group to reach the number n. This ensures that each Merkle tree is a perfect binary tree with n leaf nodes and a constant depth of log2(n).

[0091] As an example, the following summary information can be packaged into a transaction, published on the blockchain, and the current timestamp or version number t1 recorded: the total number of records in the dataset N, and the Merkle root hash list: an ordered list or mapping of the second root hash values ​​of all m data groups [Merkle_Root_0,Merkle_Root_1,...,Merkle_Root_m-1].

[0092] In some embodiments, the data provider preprocesses and groups the full data, then constructs the core structure of a Merkle tree for each data group. As shown in Figure 4, for a data group containing n records, the key field of each record is concatenated with a unique global index, and a corresponding leaf node hash (e.g., Leaf 1:Hash(data_k)) is generated through hash calculation. These leaf node hashes are paired and aggregated through multiple layers of hashing (e.g., Hash H1-H2, Hash H12-H34), ultimately converging to form a unique top-level hash value, namely the Merkle root hash (Merkle Root_i) of the group. This root hash, as a digital fingerprint of the integrity and order of the group's data, will be submitted to the blockchain as an immutable data digest along with the root hashes of other groups and the total data volume N. This provides a unique and reliable comparison benchmark for efficient and verifiable proof of existence when subsequent auditors initiate random challenges.

[0093] In some embodiments of this disclosure, a Merkle tree is generated for each group based on the global index of each data, and a second root hash value is obtained for each Merkle tree, including:

[0094] For each of the multiple groups, perform the following steps in sequence:

[0095] For each data point in the group, extract the key fields of the data, and concatenate the key fields with the global index of the data to obtain the concatenated result;

[0096] The second hash value of the concatenated result is calculated according to the preset hash calculation method;

[0097] Use all the second hash values ​​in the group as the hash values ​​of the leaf nodes to generate the Merkle tree of the group, and obtain the second root hash value of the Merkle tree.

[0098] In this embodiment of the disclosure, for each data group obtained according to the preset rules, each data record in the group can be traversed. First, its core identifier or attribute is extracted as a key field. Then, the key field is concatenated with the global index assigned to this data. A preset cryptographic hash function is applied to the concatenation result, and the generated hash value is used as the corresponding leaf node hash value of the data in the Merkle tree.

[0099] After the hashes of all leaf nodes within a group have been calculated, these hash values ​​are used as the bottom-level leaves. Following the binary tree structure of a Merkle tree, the hashes of the parent nodes are calculated in pairs from bottom to top. Through recursive aggregation, a unique top-level hash digest that condenses all the data within the group and its global index is finally generated for the group. This is the Merkle tree root hash value (second root hash value) of the group, thus ensuring that the root hash of each tree is bound to the precise content and global order of the data it contains.

[0100] In some embodiments of this disclosure, storing the second root hash value of all groups into the blockchain in step 302 may specifically include: storing the second root hash value of all groups and the total data volume information of the full data as a data digest into the blockchain.

[0101] In one embodiment, the data provider does not simply submit scattered Merkle root hashes, but rather a complete list of the second root hashes of all groups in sequence, packaged together with the total data volume (i.e., the total number of records N) to form a complete data digest. This digest is then submitted and recorded on the blockchain via a transaction. On the one hand, the ordered list of root hashes provides an explicit and locatable comparison benchmark for auditors to subsequently verify any individual data entry; on the other hand, the synchronous recording of the total data volume N on the blockchain is a public and definite commitment to the data scale, locking in the upper limit of the global index range (0 to N-1). Through the immutability and traceability of the blockchain, any tampering, omission, or misrepresentation of the total amount of the original data will lead to the failure of verification based on the on-chain digest, thus technically solidifying the integrity and scale authenticity of the data.

[0102] In some embodiments, when the full data D of data provider B changes (addition, deletion, or modification) and its data state needs to be updated, for example at a new time t2 (t2>t1), all operations in the data digest generation and on-chain process can be completely repeated to generate a completely new set of data digests (new total number of records N', new Merkle root hash list) and publish it on the blockchain as a new record of version t2.

[0103] The blockchain will retain data from all historical versions, such as t1 and t2, providing a traceable basis for audits at different points in time.

[0104] According to the data auditing method based on Merkle tree and blockchain proposed in this disclosure, by introducing a continuous and unique global index and incorporating it into hash calculation, the verification of a single piece of data can be precisely bound to the full data scale, effectively preventing the possibility of data providers cheating by hiding part of the data. The auditor only needs to complete the verification independently based on its own existing first data and the index and proof returned by the data provider, without having to provide the full data to the auditor. Thus, the audit can be completed under the premise of ensuring data sovereignty and privacy. Furthermore, by using blockchain to immutably store the Merkle tree root hash value, a reliable comparison benchmark and evidence are provided for the entire audit process, making the audit conclusion publicly verifiable, thereby establishing a solid trust mechanism in a decentralized environment.

[0105] Figure 5 is a block diagram of a data auditing device based on Merkle tree and blockchain according to an exemplary embodiment. It should be noted that the data auditing device based on Merkle tree and blockchain in this embodiment is applied to a first server. Referring to Figure 5, the device includes a sending unit 501, a calculation unit 502, and a comparison unit 503.

[0106] The sending unit 501 is used to send a data audit request including the first data to the second server, so that the second server can determine the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree based on the data audit request; the first data is the data provided by the second server to the first server; the Merkle tree is generated based on the full data and the global index of each full data; the global index of the full data is continuous and unique;

[0107] The calculation unit 502 is used to receive the global index and Merkel proof sent by the second server, calculate the first hash value of the first data based on the global index and in the same hash calculation method as the leaf hash value in the Merkel tree, and calculate the first root hash value based on the first hash value and the Merkel proof.

[0108] The comparison unit 503 is used to obtain the second root hash value corresponding to the global index from the blockchain, compare the first root hash value with the second root hash value, and if the first root hash value and the second root hash value are consistent, the data audit is confirmed to be passed; the second root hash value is the root hash value of the Merkle tree pre-stored by the second server.

[0109] In some embodiments of this disclosure, the comparison unit 503 may specifically be used for:

[0110] Based on the preset grouping method of the full data, the target group to which the global index belongs is determined; the full data corresponds to multiple Merkle trees, and each Merkle tree is generated based on the data in a group; the grouping is obtained by the second server dividing the full data according to the preset grouping method;

[0111] Obtain the root hash value of the Merkle tree corresponding to the target group from the blockchain to obtain the second root hash value.

[0112] In some embodiments of this disclosure, the apparatus may further include an acquisition unit for acquiring total data volume information of all data from the blockchain based on a global index.

[0113] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0114] According to the data auditing device based on Merkle tree and blockchain proposed in this disclosure, by introducing a continuous and unique global index and incorporating it into hash calculation, the verification of a single piece of data can be precisely bound to the full data scale, effectively preventing the possibility of data providers cheating by hiding part of the data. The auditor only needs to complete the verification independently based on its own existing first data and the index and proof returned by the data provider, without having to provide the full data to the auditor. Thus, the audit can be completed under the premise of ensuring data sovereignty and privacy. Furthermore, by using blockchain to immutably store the Merkle tree root hash value, a reliable comparison benchmark and evidence are provided for the entire audit process, making the audit conclusion publicly verifiable, thereby establishing a solid trust mechanism in a decentralized environment.

[0115] Figure 6 is a block diagram of another data auditing device based on Merkle tree and blockchain according to an exemplary embodiment. It should be noted that the data auditing device based on Merkle tree and blockchain in this embodiment is applied to a second server. Referring to Figure 6, the device includes a determining unit 601 and a sending unit 602.

[0116] The determining unit 601 is configured to respond to receiving a data audit request including first data sent by the first server, and based on the data audit request, determine the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is the data provided by the second server to the first server; the Merkle tree is generated based on the full data and the global index of each full data; the global index of the full data is continuous and unique;

[0117] The sending unit 602 is used to send the global index and Merkel proof to the first server, so that the first server calculates the first hash value of the first data based on the global index, in the same way as the leaf hash value in the Merkel tree. Based on the first hash value and the Merkel proof, the first root hash value is calculated. The second root hash value corresponding to the global index is obtained from the blockchain. The first root hash value and the second hash value are compared. If the first root hash value and the second hash value are consistent, the data audit is confirmed to be successful. The second hash value is the root hash value of the Merkel tree pre-stored by the second server.

[0118] In some embodiments of this disclosure, the apparatus may further include a storage unit, which is specifically used for:

[0119] Perform a global sort on each data point in the full dataset according to a preset sorting method;

[0120] Each data point after global sorting is assigned a separate global index; the global indices of the data in the entire dataset are consecutive and unique.

[0121] The full dataset is grouped according to a preset data volume to obtain multiple groups;

[0122] Based on the global index of each data point, a Merkle tree is generated for each group, and the second root hash value of each Merkle tree is obtained;

[0123] Store the second root hash value of all groups into the blockchain.

[0124] In some embodiments of this disclosure, the storage unit is specifically used for:

[0125] For each of the multiple groups, perform the following steps in sequence:

[0126] For each data point in the group, extract the key fields of the data, and concatenate the key fields with the global index of the data to obtain the concatenated result;

[0127] The second hash value of the concatenated result is calculated according to the preset hash calculation method;

[0128] Use all the second hash values ​​in the group as the hash values ​​of the leaf nodes to generate the Merkle tree of the group, and obtain the second root hash value of the Merkle tree.

[0129] In some embodiments of this disclosure, the storage unit is specifically used to store the second root hash value of all groups and the total data volume information of the full data as a data digest to the blockchain.

[0130] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0131] According to the data auditing device based on Merkle tree and blockchain proposed in this disclosure, by introducing a continuous and unique global index and incorporating it into hash calculation, the verification of a single piece of data can be precisely bound to the full data scale, effectively preventing the possibility of data providers cheating by hiding part of the data. The auditor only needs to complete the verification independently based on its own existing first data and the index and proof returned by the data provider, without having to provide the full data to the auditor. Thus, the audit can be completed under the premise of ensuring data sovereignty and privacy. Furthermore, by using blockchain to immutably store the Merkle tree root hash value, a reliable comparison benchmark and evidence are provided for the entire audit process, making the audit conclusion publicly verifiable, thereby establishing a solid trust mechanism in a decentralized environment.

[0132] Figure 7 is a block diagram of an apparatus for a Merkle tree and blockchain-based data auditing method according to an exemplary embodiment. For example, apparatus 700 can be an electronic device, such as a mobile phone, computer, digital broadcasting terminal, messaging device, tablet device, personal digital assistant, etc.

[0133] Referring to FIG7, the device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, and a communication component 716.

[0134] Processing component 702 typically controls the overall operation of device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0135] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0136] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 700.

[0137] Multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0138] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0139] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0140] Communication component 716 is configured to facilitate wired or wireless communication between device 700 and other devices. Device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0141] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0142] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of the device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0143] In an exemplary embodiment, a computer program product is also provided, including a computer program that implements the above-described method when executed by the processor 720 of the device 700.

[0144] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0145] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data auditing method based on Merkle trees and blockchain, characterized in that, The method, applied to a first server, includes: sending a data audit request, including first data, to a second server, so that the second server, based on the data audit request, determines the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is data provided by the second server to the first server; the Merkle tree is generated based on the full data and the respective global indexes of the full data; the global indexes of the full data are continuous and unique; receiving the global index and the Merkle proof sent by the second server, calculating the first hash value of the first data based on the global index using the same hash calculation method as the leaf hash values ​​in the Merkle tree, and based on the first hash value and... The Merkel proof involves calculating a first root hash value and retrieving a second root hash value corresponding to the global index from the blockchain. This includes: determining the target group to which the global index belongs based on a preset grouping method for the full data; the full data corresponds to multiple Merkel trees, each generated based on data in a group; the grouping is obtained by the second server dividing the full data according to the preset grouping method; retrieving the root hash value of the Merkel tree corresponding to the target group from the blockchain to obtain the second root hash value; comparing the first root hash value with the second root hash value; if the first root hash value and the second root hash value are consistent, the data audit is confirmed to be successful; the second root hash value is the root hash value of the Merkel tree pre-stored by the second server.

2. The data auditing method based on Merkle trees and blockchain according to claim 1, characterized in that, Sending the data audit request including the first data to the second server includes: randomly selecting one or more data items from all historical data provided by the second server as the first data, generating a data audit request including the first data, and sending the data audit request to the second server.

3. The data auditing method based on Merkle trees and blockchain according to claim 1, characterized in that, The method also includes: obtaining the total data volume information of the full data from the blockchain.

4. A data auditing method based on Merkle trees and blockchain, characterized in that, Applied to a second server, the method includes: responding to receiving a data audit request including first data sent by a first server; based on the data audit request, determining the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is data provided by the second server to the first server; the Merkle tree is generated based on the full data and its respective global index; the global index of the full data is continuous and unique; sending the global index and the Merkle proof to the first server, so that the first server calculates the first hash value of the first data based on the global index, using the same calculation method as the leaf hash value in the Merkle tree, and based on the first hash value and the... The Merkle proof involves calculating the first root hash value and obtaining the second root hash value corresponding to the global index from the blockchain, including: determining the target group to which the global index belongs based on a preset grouping method for the full data; the full data corresponds to multiple Merkle trees, each Merkle tree being generated based on data in a group; the grouping is obtained by the second server dividing the full data according to the preset grouping method; obtaining the root hash value of the Merkle tree corresponding to the target group from the blockchain to obtain the second root hash value; comparing the first root hash value with the second root hash value; if the first root hash value and the second root hash value are consistent, the data audit is confirmed to be successful; the second root hash value is the root hash value of the Merkle tree pre-stored by the second server.

5. The data auditing method based on Merkle tree and blockchain according to claim 4, characterized in that, The method further includes: globally sorting each data point in the full dataset according to a preset sorting method; assigning a global index to each globally sorted data point; ensuring that the global indexes of the data in the full dataset are continuous and unique; grouping the full dataset according to a preset data volume to obtain multiple groups; generating a Merkle tree for each group based on the global index of each data point to obtain the second root hash value of each Merkle tree; and storing the second root hash values ​​of all groups in the blockchain.

6. The data auditing method based on Merkle trees and blockchain according to claim 5, characterized in that, The process of generating a Merkle tree for each group based on the global index of each data point, and obtaining the second root hash value of each Merkle tree, includes: for each of the multiple groups, performing the following steps in sequence: for each data point in the group, extracting the key field of the data, concatenating the key field with the global index of the data to obtain the concatenation result; calculating the second hash value of the concatenation result according to a preset hash calculation method; using all the second hash values ​​in the group as the hash values ​​of the leaf nodes to generate the Merkle tree of the group, and obtaining the second root hash value of the Merkle tree.

7. The data auditing method based on Merkle tree and blockchain according to claim 5, characterized in that, The step of storing the second root hash value of all groups into the blockchain includes: storing the second root hash value of all groups and the total data volume information of the full data as a data digest into the blockchain.

8. A data auditing device based on Merkle trees and blockchain, characterized in that, Applied to a first server, the system includes: a sending unit, configured to send a data audit request including first data to a second server, so that the second server, based on the data audit request, determines the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is data provided by the second server to the first server; the Merkle tree is generated based on the full data and the respective global indexes of the full data; the global indexes of the full data are continuous and unique; and a calculation unit, configured to receive the global index and the Merkle proof sent by the second server, and calculate the first hash value of the first data based on the global index, using the same hash calculation method as the leaf hash values ​​in the Merkle tree, and based on the first hash value... The first hash value is calculated using the hash value and the Merkel proof. A comparison unit is used to obtain the second hash value corresponding to the global index from the blockchain, including: determining the target group to which the global index belongs based on a preset grouping method for the full data; the full data corresponds to multiple Merkel trees, each Merkel tree being generated based on data in a group; the grouping is obtained by the second server dividing the full data according to the preset grouping method; obtaining the root hash value of the Merkel tree corresponding to the target group from the blockchain to obtain the second hash value; comparing the first hash value with the second hash value; if the first hash value and the second hash value are consistent, the data audit is confirmed to be successful; the second hash value is the root hash value of the Merkel tree pre-stored by the second server.

9. A data auditing device based on Merkle trees and blockchain, characterized in that, An application to a second server includes: a determining unit, configured to, in response to receiving a data audit request including first data sent by a first server, determine, based on the data audit request, the global index of the first data in the full data of the second server, and the Merkle proof of the global index in its corresponding Merkle tree; the first data is data provided by the second server to the first server; the Merkle tree is generated based on the full data and the respective global indexes of the full data; the global indexes of the full data are continuous and unique; and a sending unit, configured to send the global index and the Merkle proof to the first server, so that the first server calculates a first hash value of the first data based on the global index, using the same calculation method as the leaf hash values ​​in the Merkle tree, based on the... The calculation of the first hash value and the Merkel proof yields the first root hash value, and the acquisition of the second root hash value corresponding to the global index from the blockchain includes: determining the target group to which the global index belongs based on a preset grouping method for the full data; the full data corresponds to multiple Merkel trees, each Merkel tree being generated based on data in a group; the grouping is obtained by the second server dividing the full data according to the preset grouping method; the root hash value of the Merkel tree corresponding to the target group is acquired from the blockchain to obtain the second root hash value; the first root hash value and the second hash value are compared, and if the first root hash value and the second hash value are consistent, the data audit is confirmed to be successful; the second hash value is the root hash value of the Merkel tree pre-stored by the second server.

Images