A continuous audit solution for big data integrity in decentralized storage

Abstract

The present invention belongs to the field of data integrity auditing, and specifically relates to a continuous auditing scheme for big data integrity in decentralized storage. It includes a data owner and a storage provider, wherein the data owner is a verifier and the storage provider is a prover. The steps of the scheme are as follows: the data owner preprocesses the data and generates public parameters. The data owner uses the data and public parameters to generate a label set, and sends the public parameters to the storage provider. The storage provider uses a continuous verifiable delay function and a retrievability proof technology and a verifiable random function to calculate through the data and public parameters to obtain a result set, a proof set and a challenge set. The generated result set is then sent to the data owner. The data owner performs verification calculations on the received result set to obtain a verification result. The present invention not only ensures that accurate verification of data integrity can be performed continuously, but also reduces the transmission costs of the challenge stage and the proof stage.

Description

Technical Field

[0001] The present invention belongs to the field of data integrity auditing, and specifically designs a data integrity auditing scheme based on a continuous verifiable delay function (cVDF), proof of retrievability (PoR) and a verifiable random function (VRF). Background Art

[0002] In recent years, data from various sources such as personal devices, the Internet, and the Internet of Things has exploded. Outsourcing storage services has become a new trend for data owners to upload big data to remote storage servers, among which cloud storage services have become a popular choice for data owners. As an integral part of cloud computing, cloud storage makes data storage a service in which data is outsourced to cloud servers maintained by cloud storage providers. However, cloud storage has a centralized storage form, and single points of failure occur due to system failures, power outages, and even malicious attacks, resulting in a large amount of data leakage or loss in cloud storage. Decentralized storage is considered to offer many attractive advantages over cloud storage. First, decentralized storage has superior elastic expansion characteristics in many cases, allowing data owners' data to be redundantly distributed to many different storage providers. Second, decentralized storage can provide users with abundant storage capacity at a lower cost. Decentralized storage aggregates storage space from many personal devices and eliminates many burdens of maintaining data centers, such as the purchase, placement, management, and cooling of servers, thereby greatly reducing storage costs. Finally, decentralized storage can collect abandoned storage in personal computers and has the potential to provide a larger storage market than cloud storage.

[0003] Although decentralized storage offers a long list of benefits, there are still various obstacles before decentralized storage is widely deployed. Since decentralized storage encourages all individuals to participate, there must be some unqualified or dishonest storage providers who provide poor storage service quality instead of the claimed storage service quality. As a result, data owners will be deceived and overcharged for the claimed storage service. In addition, unqualified storage providers usually have limited computing power, aging hard drives, low-version operating systems, etc., which are prone to security risks and threaten the data owner's data. Therefore, continuously detecting data integrity throughout the storage cycle to detect storage service quality has become a crucial and challenging issue in decentralized storage.

[0004] In order to ensure the uniqueness and anti-parallelism of the calculation results, predecessors have proposed many types of verifiable delay functions (VRFs), including general verifiable delay functions (VDF), verifiable delay functions with backdoors (Trapdoor VDF, TDF), and decodable verifiable delay functions (Decodable VDF). In 2020, Ephraim et al. proposed a continuous verifiable delay function cVDF (Naomi Ephraim, Cody Freitag, Ilan Komargodski, et al. Continuous Verifiable Delay Functions [J]. Springer, Cham, 2020.). Compared with other verifiable delay functions, cVDF can continuously verify the calculation results during the calculation process, that is, the existing results can be used to continue the calculation after the data calculation is interrupted. cVDF adopts the structure of an n-ary tree, which expresses the calculation process of the data in the sequential calculation of the leaf nodes, and uses the characteristics of node merging in the tree to simplify the result data, thereby reducing the time for subsequent verification and increasing the time span of each verification. The nodes mentioned below refer to the nodes on the n-ary tree structure in cVDF. The calculation of each node includes Eval function and Verify function. cVDF includes initialization function, Eval function and Verify function. In addition, according to the existing experimental data, when the width of the tree structure is 64, the total number of nodes is 534306, and the total calculation time of the Eval function of all nodes in cVDF is about 1 hour.

[0005] In the scheme based on big data, there is often a step of randomly extracting data. In order to resolve the controversy over the sampling results, the predecessors proposed the verifiable random function VRF (S. Goldberg, L. Reyzin, D. Papadopoulos, and J. Vcelak, "Verifiable random functions (vrfs)," Tech. Rep., Accessed: Jun. 06, 2020. [Online]. Available: https: / / tools.ietf.org / id / draft-irtf-cfrg-vrf-07.html.), so that the correctness of the sampling results can be concluded through the verification function of VRF. The VRF technology includes the initialization function, the hash function and the verify function.

[0006] When users store data in a remote server, in order to check the integrity of the continuous data in the remote server, predecessors have proposed technologies such as Provable Data Possession (PDP) and Proof of Retrievability (PoR). Most of these schemes are based on homomorphic verifiable tags, enabling data owners or third parties to verify whether the data stored in the storage provider is complete. The PoR technology includes initialization function, Store function, Challenge function, Prove function and Verify function. However, in the above schemes, frequent checks on storage providers will bring a lot of communication and computing costs to the verifier, and the checking process requires the verifier to be in an interactive state all the time. To address this problem, Atennies et al. proposed a verification scheme PoSt (Ateniese, G., Burns, R., et al., 2007. Provable data possession at untrusted stores. In: Proceedings of the 14th ACM Conference on Computer and Communications Security. ACM, Scottsdale, Arizona, USA, pp. 598–609.), which effectively checks the continuous integrity and availability of each single file without the need for frequent interaction with the verifier, greatly reducing the communication and computing costs of the challenge and verification phases. However, their scheme still incurs a lot of communication and computing overhead when verifying multiple data files, especially when considering big data scenarios. In big data scenarios, a large number of data files are frequently generated and outsourced to decentralized storage. For most data owners who usually do not have unlimited communication and computing resources, the entire computing phase is a huge challenge.

[0007] To sum up, data integrity auditing in the context of big data plays an increasingly important role in data processing. The existing verification algorithms at home and abroad still have shortcomings and deficiencies. It is necessary to invent a new data integrity verification algorithm. This algorithm not only needs to be able to continuously verify the integrity of massive data, but also needs to ensure the transmission speed and overall computing speed, so as to achieve the purpose of efficient and continuous data integrity verification. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a data integrity verification method based on continuous verifiable delay function (cVDF), proof of retrievability (PoR) and verifiable random function (VRF). The cVDF technology in the scheme can ensure continuous verification of data integrity, the sampling strategy ensures that the data owner greatly reduces time overhead, the VRF ensures that the randomness of sampling can be verified, and the PoR technology ensures that the integrity of the data can be verified. Therefore, under the premise of ensuring accurate verification of data integrity, this scheme can more efficiently and continuously verify the integrity of big data.

[0009] The solution of the present invention to solve the above technical problems is:

[0010] A data integrity verification method based on continuous verifiable delay function (cVDF), proof of retrievability (PoR) and verifiable random function (VRF), as shown in the attached Figure 1 As shown. The scheme involves two parties, namely the data owner and the storage provider. The method includes four stages, namely the initialization stage, the store stage, the challenge stage, the proof stage, and the verification stage.

[0011] Before the scheme starts calculating, the data owner uploads the data D stored by itself to the storage provider, and the storage provider saves the data copy D'. The data D here has the same content as the data copy D', that is, the data D' whose integrity is to be verified. The data D is a collection of files F, and each file F has a unique ID. The sampling of time in this scheme refers to the sampling of the data that has been calculated. Since the calculation of data takes time, the sampling of the calculated data is the sampling of the past time.

[0012] First, in the initialization phase, the data owner preprocesses the data D and generates public parameters {pp}. Secondly, in the Store phase, the data owner uses the data D and public parameters {pp} to generate the hash value {Hash} and tag set {Tag} of the data. In the Challenge phase, the data owner uses the data D and public parameters {pp} to generate a challenge set {C}, and sends the public parameters {pp} and challenge set {C} to the storage provider. Then, in the Prove phase, the storage provider uses the data copy D' and public parameters {pp} to perform calculations and extractions to obtain the result set {Y}, the proof set {π}, the random number set {R}, and the file number id set, and sends them to the data owner. Finally, in the Verify phase, the data owner performs verification calculations to obtain the verification results. As shown in the attached figure Figure 1 shown.

[0013] The calculation steps of each stage are described in detail below.

[0014] Initialization phase:

[0015] 1 The data owner sets the required storage time T, and calculates and sets the total number of nodes k in cVDF based on the time overhead of calculating a single node in cVDF. The data owner sets the number of extracted files m based on the verification probability and verification duration in the Verify phase, which is generally set to 20.

[0016] 2 Based on the given total number of nodes k, the data owner obtains the public parameters {pp_cvdf} of cVDF through the initialization function of cVDF.

[0017] 3. Set the number of bits λ of the private key in the PoR algorithm. The data owner obtains the public parameter {pp_por} of PoR through the PoR initialization function.

[0018] 4 The data owner obtains the public parameter {pp_vrf} of VRF through the initialization function of VRF.

[0019] 5 Package the parameters k, m, {pp_cvdf}, {pp_por} and {pp_vrf} and set them as the public parameter {pp}.

[0020] Store stage:

[0021] According to the public parameters {pp} and data D, the data owner calculates and generates the hash value set {Hash} and tag set {Tag} of data D through the Store function of PoR.

[0022] Challenge stage:

[0023] 1 Based on the public parameters {pp}, the data owner calculates and generates the challenge set {C} through the Challenge function of PoR.

[0024] 2. Send the public parameters {pp} and the challenge set {C} to the storage provider.

[0025] Prove stage:

[0026] 1. Through the public parameter {pp}, the storage provider uses the Eval function of cVDF to generate the result set {y} and the proof set {π} of the cVDF calculation, and records the current y k and π k , and continue to calculate the Eval function in cVDF.

[0027] 2 Storage providers are provided by using y k , randomly select the result set {y} and the proof set {π}, a total of m, and obtain the extracted result set {ym} and prove the set {π m}.

[0028] 3. Through the public parameter {pp}, the storage provider uses the VRF hash function to hash the result set {y m} and prove the set {π m} to calculate and generate the corresponding random number set {r m} and prove the set {π' m}.

[0029] 4 The storage provider uses the public parameters {pp} and the random number set {r m}, randomly sample the data copies D', and obtain the extracted file set {F m}.

[0030] 5 The storage provider uses the Prove function of PoR to verify the public parameters {pp} and the extracted file set {F m} respectively, and obtain the proof of data integrity {π" m}.

[0031] 6 The result set {y m}、Random number set {r m}、File Set {F m}, the set of IDs and the set of proofs of cVDF {π m}、VRF proof set {π' m} and the proof set of PoR {π” m}Sent to the data owner.

[0032] This stage is as attached Figure 2 shown. Figure 2 The tree structure is formed by the cVDF initialization phase. The squares in the figure represent parameters or files; the circles represent each calculation in the cVDF. The gray circle indicates that the data of the event segment has been calculated and merged through the cVDF Eval function. The black circle indicates the currently retained calculation result, and the white circle indicates that it has not been calculated. The rectangle represents the calculation of the function, in which the arrows pointing in and out represent the input of parameters and the output of calculation results respectively; the dotted line represents the extraction of the current file.

[0033] Verify stage:

[0034] 1 The data owner, as the verifier, sends the result set {y m} and prove the set {π m} as a parameter and put it into the Verify function of cVDF for verification calculation. If the function returns 1, the verification is successful and the process continues; otherwise, the verification is terminated and the verification failure information is sent to the storage provider.

[0035] 2 The data owner, as the verifier, sends the random number set {r m} and prove the set {π' m} as a parameter and put it into the Verify function of VRF for verification calculation. If the function returns 1, the verification is successful and the process continues; otherwise, the verification is aborted and the verification failure information is sent to the storage provider.

[0036] 3 The data owner, as the verifier, sends the file set {F m}'s numbered id set and proof set {π" m} as a parameter and put it into the Verify function of PoR for verification calculation. If the function returns 1, the verification is successful, the current verification phase ends, and the verification of the next process of the data owner continues. Otherwise, the verification failure information is sent to the storage provider.

[0037] Beneficial Effects

[0038] 1. Through random sampling verification of time, the overall efficiency can be improved and the overall time cost can be reduced while ensuring the fairness of sampling.

[0039] 2 When storing data indefinitely, the integrity of the data is continuously verified through the result set and proof set of the storage provider. Even if the storage provider is changed in the middle, there is no need to start the calculation from scratch, because the previous result set and proof set can still be effectively verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 , one Secondary verification process flow chart;

[0041] Figure 2 , Prove phase schematic diagram. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with an embodiment. This embodiment specifically describes a data integrity audit solution process based on a continuous verifiable delay function (cVDF), proof of retrievability (PoR) and a verifiable random function (VRF). The process completes time sampling, continuous calculation verification, random sampling verification and continuous data integrity verification.

[0043] Experimental data show that when the width of the tree structure is 64 and the total number of nodes is 534306, the total computational time of the Eval function in cVDF is about 1 hour. The data owner stores the existing data with a term of 100 years (i.e., indefinitely) to the storage provider. For the tree width in cVDF, the data owner sets it to 64 by default, and the total number of leaf nodes k is set to 4680520560.

[0044] The data has now been stored for two years. During these two years, the data owner has verified the integrity of the stored data according to the following protocol. The current data owner will conduct the nth verification, which includes the initialization phase, the store phase, the challenge phase, the prove phase, and the verify phase, as shown below.

[0045] Initialization phase:

[0046] 1 Set the default number of extracted files m to 20.

[0047] 2 According to the above structure, the data owner obtains the public parameters {pp_cvdf} of cVDF through the initialization function of cVDF.

[0048] 3. Set the number of bits of the private key in the PoR algorithm to 1024 to ensure the security of the key. The data owner obtains the public parameters {pp_por} of PoR, that is, the public key and private key of PoR, through the initialization function of PoR.

[0049] 4 The data owner obtains the VRF public parameter {pp_vrf} through the VRF initialization function.

[0050] 5 Package the parameters k, m, {pp_cvdf}, {pp_por} and {pp_vrf} and set them as the public parameter {pp}.

[0051] Store stage:

[0052] According to the public parameters {pp} and data D, the data owner reads each file in data D in turn, and then calculates and generates each file d through the PoR's Store function i Hash value i and tag i . The results of each file are packaged into a hash value set {Hash} and a tag set {Tag}. In order to simulate the real situation, the data D consists of 200 64MB files.

[0053] Challenge stage:

[0054] 1 Based on the public parameters {pp}, the data owner calculates and generates the challenge set {C} through the Challenge function of PoR.

[0055] 2. Send the public parameters {pp} and the challenge set {C} to the storage provider.

[0056] Prove stage:

[0057] 1. Through the public parameter {pp}, the storage provider uses the Eval function of cVDF to generate the result set {y} and the proof set {π} of the cVDF calculation, and records the current y k and π k , and continue to calculate the Eval function in cVDF.

[0058] 2 Storage providers are provided by using y k , randomly select the result set {y} and the proof set {π}, a total of m, and obtain the extracted result set {y m} and prove the set {π m}.

[0059] 3. Through the public parameter {pp}, the storage provider uses the VRF hash function to hash the result set {y m} and prove the set {π m} to calculate and generate the corresponding random number set {r m} and prove the set {π' m}.

[0060] 4 The storage provider uses the public parameters {pp} and the random number set {r m}, randomly sample the data copies D', and obtain the extracted file set {F m}.

[0061] 5 The storage provider uses the Prove function of PoR to verify the public parameters {pp} and the file set {F m} respectively, and obtain the proof of data integrity {π" m}.

[0062] 6 The result set {y m}、Random number set {r m}、File Set {F m}, the set of IDs and the set of proofs of cVDF {π m}、 VRF proof set {π' m} and the proof set of PoR {π” m}Sent to the data owner.

[0063] This stage is as attached Figure 2 shown. Figure 2The tree structure is formed by the initialization phase of cVDF. The squares in the figure represent parameters or files; the circles represent each calculation in cVDF. The gray circle indicates that the data of the event segment has been calculated and merged through the Eval function of cVDF. The black circle indicates the currently retained calculation result, and the white circle indicates that it has not been calculated; the rectangle represents the calculation of the function, in which the arrows pointing in and out represent the input of parameters and the output of calculation results respectively; the dotted line represents the extraction of the current file. The figure shows that only one file is extracted, but multiple files can also be extracted. This will not affect other structures.

[0064] In the tree structure, the public parameter {pp} is put into the leftmost node as the initial parameter to calculate the Eval function of cVDF, and the result y1 and proof π1 are obtained. Then the next node is calculated or the subtree is merged according to the cVDF technology. When the user needs to check the data integrity, the result y of the kth node calculated at the current moment is extracted. k And prove π k , that is, the result y of the rightmost black leaf node k And prove π k . And the result y k As a random seed, sample other nodes that have completed the calculation to obtain the sampled result set {y} and proof set {π}. Then use each result yi as a random seed and put it into VRF. After the VRF hash function is used to calculate the random number r i and randomness proof set π' i . Then by r i As a random seed, sample the file set {F} and use the sampled F i Put it into the Prove function of PoR for calculation to get the integrity proof π" i .

[0065] Verify stage:

[0066] 1 The data owner, as the verifier, sends the result set {y m} and prove the set {π m} as a parameter and put it into the Verify function of cVDF for verification calculation. If the function returns 1, the verification is successful and the process continues; otherwise, the verification is terminated and the verification failure information is sent to the storage provider.

[0067] 2 The data owner, as the verifier, sends the random number set {r m} and prove the set {π' m} as a parameter and put it into the Verify function of VRF for verification calculation. If the function returns 1, the verification is successful and the process continues; otherwise, the verification is aborted and the verification failure information is sent to the storage provider.

[0068] 3 The data owner, as the verifier, sends the file set {F m}'s numbered id set and proof set {π" m} as a parameter and put it into the Verify function of PoR for verification calculation. If the function returns 1, the verification is successful, the current verification phase ends, and the verification of the next process of the data owner continues. Otherwise, the verification failure information is sent to the storage provider.

[0069] At this point, the data owner has completed the nth verification. Before the data reaches the storage period, the data owner can perform the n+1th verification at any time.

[0070] In traditional data integrity auditing solutions, data owners only audit individual files one by one, which takes too much time for a single audit. In the context of big data, our node sampling and file sampling aggregation methods in the Prove phase greatly reduce the time cost of auditing and improve verification efficiency. The sampling of nodes is to perform a single audit on the result set {y m} and prove the set {π m Due to the uncertainty of the number of files and the change of storage time, this paper does not compare the efficiency with the traditional solution, and only gives the experimental results of this solution.

[0071] Table 1 shows the time overheads of the Store, Prove, and Verify phases when m=20 and k=64, and the leaf nodes are 63, 4095, 262143, and 166777215, respectively. Obviously, when the leaf nodes grow exponentially, the time overhead of the Store phase shows a stable trend, and the time overhead of the Prove and Verify phases also show an exponential growth trend. The unit of measurement for the time overhead is seconds.

[0072] Table 1 Time cost of each stage when leaf nodes are different

[0073] .

Claims

1. A method for continuous auditing the integrity of big data in decentralized storage is used to perform integrity audit on a data copy D' stored by a storage provider. The data copy D' is a copy of the data D stored by the data owner itself. The data copy D' is composed of multiple files {F} and each file has a unique number id. The cVDF adopts an n-ary tree structure, which is characterized by: Random sampling is performed on the time level of the continuous calculation process, that is, the data result set that has been calculated, and the sampling results are used to calculate and verify the data integrity; the randomness of the sampled data is verified by the verifiable random function VRF, and the integrity of the stored data is continuously audited; it specifically includes five stages, namely the initialization stage, the Store stage, the Challenge stage, the Prove stage, and the Verify stage; among them, the initialization stage is used to preprocess the data D and generate public parameters {pp}, the Store stage is used to generate the hash value {Hash} and the tag set {Tag} of the data, that is, to generate the unique identifier of the data, the Challenge stage is used to generate the challenge set {C}, and send the public parameters {pp} and the challenge set {C} to the storage provider, the Prove stage is used to generate the result set {Y}, the proof set {π}, the random number set {R} and the file number id set, and send them to the data owner, and the Verify stage is used to verify the data stored by the storage provider to obtain the verification result; The initialization phase is as follows: 1) The data owner sets the required storage time T, and calculates and sets the total number of nodes k in the cVDF according to the node duration consumption; the data owner sets the number of extracted files m according to the verification probability and verification duration of the subsequent Verify phase; 2) Based on the given total number of nodes k, the data owner calculates through the cVDF initialization function to generate the public parameters {pp_cvdf} required for VDF; 3) Set the number of digits λ of the private key in the PoR algorithm. The data owner calculates the public parameter {pp_por} required for generating PoR through the PoR initialization function; 4) The data owner calculates through the VRF initialization function to generate the public parameter {pp_vrf} required for VRF; 5) Pack the parameters k, m, {pp_cvdf}, {pp_por} and {pp_vrf} and set them as the public parameter {pp}; The Store phase is as follows: According to the public parameters {pp} and data D, the data owner calculates through the PoR's Store function to generate the hash value set {Hash} and tag set {Tag} of data D; The Challenge stage is as follows: 1) Based on the public parameters {pp}, the data owner calculates through the Challenge function of PoR to generate a challenge set {C}; 2) The data owner sends the public parameters {pp} and the query set {C} to the storage provider; The Prove stage is as follows: 1) Through the public parameter {pp}, the storage provider uses the Eval function of cVDF to calculate, Generate the result set {y} and proof set {π} of cVDF calculation, {y}={y 1、 …、y k }, {π}={π 1、 …、π k }, where y k represents the result at the current time k, proving π k Represents the proof of the current moment k, continues to calculate the Eval function in cVDF, and continuously updates the result set {y} and the proof set {π}; 2) Storage providers will k As the seed for random extraction, randomly extract the result set {y} and the proof set {π}, extract m in total, and obtain the extracted result set {y m } and prove the set {π m }; 3) Through the public parameter {pp}, the storage provider uses the VRF hash function to hash the result set {y m } and prove the set {π m } to perform calculations, and after the calculations are completed, the corresponding random number set {r m } and prove the set {π' m }; 4) The storage provider uses the public parameters {pp} and the random number set {r m }={r1,…,r m }, where the public parameter {pp} and the random number r are used i Randomly sample the data copy D', repeat the sampling m times, and only extract one file each time to obtain the extracted file set {F m }; 5) The storage provider uses the Prove function of PoR to verify the public parameters {pp}, the challenge set {C} and the extracted file set {F m } respectively; specifically, the extracted file F i , public parameters {pp} and challenge set {C} as parameters, put them into the Prove function of PoR for calculation, and after the calculation is completed, π is obtained" i , until the extracted file F m It is also calculated by the Prove function of PoR to obtain the proof of the integrity of the calculated data {π” m }; 6) The result set {y m }、Random number set {r m }、File Set {F m }, the set of IDs and the set of proofs of cVDF {π m }、VRF proof set {π' m } and the proof set of PoR {π” m }Send to data owner; The Verify stage is as follows: 1) The data owner, as the verifier, sends the result set {y m } and prove the set {π m } as a parameter and put it into the Verify function of cVDF for verification calculation. If the function returns 1, the verification is successful and the process continues; otherwise, the verification is terminated and the verification failure information is sent to the storage provider; 2) The data owner, as the verifier, sends the random number set {r m } and prove the set {π' m } as a parameter and put it into the Verify function of VRF for verification calculation. If the function returns 1, the verification is successful and the process continues; otherwise, the verification is terminated and the verification failure information is sent to the storage provider; 3) The data owner, as the verifier, sends the file set {F m }'s numbered id set and proof set {π" m } as a parameter and put it into the Verify function of PoR for verification calculation. If the function returns 1, the verification is successful, the current verification phase ends, and the verification of the next process of the data owner continues. Otherwise, the verification failure information is sent to the storage provider; Among them, cVDF includes initialization function, Eval function and Verify function, PoR includes initialization function, Store function, Challenge function, Prove function and Verify function, and VRF includes initialization function, Hash function and Verify function.

2. The method for continuous auditing of big data integrity in decentralized storage according to claim 1, characterized in that: Random sampling is performed on the time level of the continuous calculation process, that is, the data result set that has been calculated, and the sampling results are used to calculate and verify the data integrity; the randomness of the sampled data is verified by the verifiable random function VRF, and the integrity of the stored data is continuously audited; When the width of the tree structure in the cVDF initialization function is 64, the total number of nodes is 534306, and the calculation of the cVDF Prove function takes 1 hour.

Images