Data management method for big data service

Abstract

The invention discloses a data management method for big data service, which relates to the field of data processing, and comprises the following steps: slicing encrypted data by adopting data slicing, and dispersedly storing the sliced data on a plurality of storage nodes; a heterogeneous multi-chain platform is set, and data processing tasks are distributed to different fragment chains in the heterogeneous multi-chain platform to be executed in parallel; deploying a verifiable delay function VDF in each fragment chain as one of consensus algorithms of the fragment chain, and selecting other consensus algorithms except the VDF algorithm according to respective service requirements of each fragment chain; the service logic of each fragment chain is deployed on the corresponding fragment chain in the form of an intelligent contract, and a cross-chain communication protocol is utilized to perform data interaction between cross-fragment chains; the calculation process of all subtasks is completed through the MPC protocol, and a global calculation result of data processing is obtained under the condition that original data is not leaked. Aiming at low data processing efficiency in the prior art, the calculation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data management method for big data services. Background Art

[0002] At present, big data processing mainly uses centralized computing frameworks such as Hadoop and Spark, which have the following limitations: First, the computing efficiency is heavily dependent on the performance of a single cluster, which is difficult to cope with the rapid growth of data scale and computing complexity; second, there is a lack of effective privacy protection measures, and user sensitive data faces a greater risk of leakage during uploading and processing; third, the data lacks a traceable and verifiable ownership mechanism, which is easy to cause integrity problems such as data falsification and denial; fourth, the system's horizontal expansion capability is insufficient, and it is impossible to fully utilize the idle computing power of the network, resulting in a waste of resources. These problems have seriously restricted the release and application of big data value.

[0003] As a decentralized, highly reliable, and tamper-resistant distributed ledger technology, blockchain provides an opportunity to build a new model of reliable and efficient big data processing. Deeply integrating blockchain with big data, using cryptographic principles to achieve privacy protection and ownership management throughout the data life cycle, and using consensus mechanisms to achieve distributed execution and verification of computing tasks with the participation of multiple parties across the network, is expected to significantly improve the efficiency and credibility of big data processing.

[0004] However, the existing blockchain system still faces two major challenges in supporting big data computing: First, the lack of verifiability. Traditional blockchain consensus, such as PoW and PoS, can only verify the legitimacy of transactions, but cannot verify the correctness of the calculation results of the data on the chain. There are risks such as node maliciousness and data tampering. Second, the computing efficiency is low. To ensure that the calculation results of the blockchain can be verified, each node needs to perform the same task and reach a consensus, resulting in a large amount of redundant calculations. In addition, the communication overhead of the consensus process is large, which seriously restricts the computing performance. This makes it difficult to balance verifiability and computing efficiency. Summary of the invention

[0005] In response to the problem of low data processing efficiency in the prior art, the present application provides a data management method for big data services. By building a heterogeneous multi-chain platform, data processing tasks are decomposed into sub-tasks for parallel processing, integrating VDF and multiple consensus mechanisms, and using a secure multi-party computing protocol to complete sub-task calculations through multi-party collaboration, thereby improving computing efficiency while ensuring verifiability.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The present application provides a data management method for big data services, including: collecting data to be processed; recording the process of collecting data in the form of transactions on the blockchain, and using zero-knowledge proof to verify the authenticity of the data under the premise of protecting privacy; homomorphically encrypting the data after authenticity verification, and then slicing the encrypted data using data slicing, and distributing the sliced ​​data on multiple storage nodes; setting up a heterogeneous multi-chain platform, and allocating the processing tasks of the stored data to different shard chains in the heterogeneous multi-chain platform for parallel execution; at the same time, deploying a verifiable delay function VDF in each shard chain as one of the consensus algorithms of the shard chain, and according to the business needs of each shard chain, in While selecting the VDF consensus algorithm, select other consensus algorithms in addition to the VDF algorithm; deploy the business logic of each shard chain on the corresponding shard chain in the form of a smart contract, and use the cross-chain communication protocol to interact with data across shard chains; use multi-party secure computing MPC to decompose the data processing tasks assigned to each distribution chain into multiple subtasks, and assign the subtasks to multiple data owners; each data owner first secretly shares its input data after homomorphic encryption, and then only executes the assigned subtask locally; finally, complete the calculation process of all subtasks through the MPC protocol, and obtain the global calculation results of data processing without leaking the original data of each data owner.

[0008] Among them, the data processing tasks are first decomposed and assigned to different shard chains to achieve parallel execution at the shard level. Each shard chain can select a suitable consensus algorithm (including but not limited to VDF) according to its own business needs and deploy corresponding business logic. The data processing tasks on each shard chain are decomposed for the second time, and the tasks of each shard chain are further decomposed into multiple subtasks, and assigned to multiple data owners for local execution to achieve parallel execution at the task level. MPC technology is used here to ensure that the execution of subtasks is completed under the premise of protecting the data privacy of all parties. After all subtasks are executed in step S5, the local calculation results of each data owner are securely aggregated through the MPC protocol to obtain the calculation results of the tasks of each shard chain, and finally the calculation results of all shard chains are summarized to obtain the global calculation results.

[0009] Furthermore, the process of collecting data is recorded on the blockchain in the form of a transaction, and zero-knowledge proof is used to verify the authenticity of the data under the premise of protecting privacy, including: recording the process of collecting data in the form of a transaction on the blockchain, the transaction contains the metadata of the time, place and operator of data collection, and the hash value of the data, but does not contain the original data; the data provider uses the zero-knowledge proof protocol to generate a data authenticity certificate for the collected original data; the data authenticity certificate is used to prove that the original data held by the data provider is consistent with the data corresponding to the hash value recorded on the blockchain, and the data collection process meets predefined requirements; the data user obtains the data authenticity certificate and the transaction information recorded on the blockchain, and by verifying the data authenticity certificate, without directly obtaining the original data, based on the metadata and hash value contained in the transaction information, confirms the authenticity of the original data held by the data provider and the compliance of the data collection process; after the verification is passed, the data provider will homomorphically encrypt the verified original data to obtain ciphertext data, and pass the ciphertext data to the data user.

[0010] Furthermore, data slicing is used to slice the encrypted data, and the sliced ​​data is dispersedly stored on multiple storage nodes, including: homomorphically encrypting the original data after authenticity verification to obtain encrypted data, and using the encrypted data as a large secret S; selecting a positive integer t as a threshold for secret sharing, and a positive integer n greater than t as the total number of shards, randomly selecting a t-1 degree polynomial f(x) on a finite field GF(p), such that f(0) = S, the coefficients of the polynomial f(x) are randomly selected t-1 elements, and the polynomial f(x) is expressed as: f(x) = a 0 +a 1 x+a 2 x 2 +......+a t-1 x t-1 modp, where a 0 =S,a 1 to a t-1 are randomly selected coefficients; x is the independent variable, p is the modulus, and t is the degree of the polynomial. i (i=1,2,......,n), calculate y i =f(x i ), get n fragments (x i ,y i ), each shard contains an x ​​value and a y value; using the threshold signature mechanism, the signing parties consisting of multiple storage nodes jointly generate a public-private key pair (p k ,s k ), public key p k Public, private keysk Use secret sharing algorithm to split into n private key shards s ki , each storage node holds a private key shard; using distributed storage, the obtained n data shards and n private key shards are dispersed and stored on multiple storage nodes, and each storage node stores a data shard (x i ,y i ) and a private key shard s ki ; Among them, the allocation of shards is random; the data user randomly selects at least t nodes from multiple storage nodes according to the threshold t of secret sharing, and collects t data shards (x i ,y i ), and according to the (x i ,y i ) value, reconstruct the t-1 degree polynomial f(x) through the Lagrange interpolation formula, take f(0) as the secret S, and get the encrypted data; the data user selects these t nodes as the signing parties from the t storage nodes of the collection shard, and uses the threshold signature algorithm. The t signing parties respectively use their private key shards sk_i to perform threshold signature on the restored encrypted data S, generate t partial signatures, and aggregate the t partial signatures to generate a complete signature to verify the validity of the complete signature; after the signature verification is passed, the decryption function of the homomorphic encryption algorithm is used to decrypt the restored encrypted data S to obtain the original data in plain text.

[0011] Furthermore, a heterogeneous multi-chain platform is set up to distribute the processing tasks of the stored data to different shard chains in the heterogeneous multi-chain platform for parallel execution, including: constructing a directed acyclic graph DAG of the subtasks to be processed, each node in the DAG corresponds to a subtask, and the directed edges represent the data dependency relationship between the subtasks; according to the number of subtask nodes in the DAG, creating shard chain instances at least equal to the number of subtasks to form a heterogeneous multi-chain platform; recording the mapping relationship between subtasks and shard chains using a distributed hash table DHT, using the subtask ID as the key and the shard chain ID as the value to construct a mapping table from tasks to shard chains; wherein the subtask ID is generated by the unique identifier of the corresponding subtask node in the DAG, and the shard chain ID is generated by the unique identifier of the corresponding shard chain instance; traversing the DAG graph to obtain the predecessor task node set of the current subtask node, the predecessor task node is the source of input data for the current subtask node, and the predecessor task node is recorded in the DHT mapping table. The computing power information of the sharding node is used to select the sharding chain with the best computing power for the current subtask node, and the sharding chain deployment information of the current subtask node is written into the DHT mapping table; the processing logic of each subtask is converted into smart contract code, and based on the sharding chain information determined by the DHT mapping table, the smart contract code is deployed to the smart contract running environment of the corresponding sharding chain; when the smart contract is executed, the storage location of the input data shard is queried according to the DHT mapping table, and a cross-chain data read request is initiated; in the consensus mechanism of each sharding chain, the verifiable delay function VDF is fixed as one of the consensus algorithms; according to the computing power of the sharding chain, at least one consensus algorithm is selected from the proof of work PoW, proof of stake PoS, delegated equity proof DPoS and Byzantine fault tolerance PBFT, and combined with VDF to form a hybrid consensus mechanism; for subtasks with data dependencies, after the upstream subtask completes the calculation, the calculation result is sent to the sharding chain where the downstream subtask is located through the cross-chain communication protocol.

[0012] Furthermore, a directed acyclic graph DAG of the subtasks to be processed is constructed, including: dividing the collected, verified and encrypted ciphertext data to be processed into multiple data shards, each data shard containing a portion of ciphertext data, and using a hash function to generate a hash value for the content of each data shard as the unique identifier of the data shard, recorded as the shard ID; based on the data shards, a MAP stage subtask is constructed, all data shards are traversed, and a corresponding MAP subtask is created for each data shard, and the data shard is used as the input data of the MAP subtask; a unique 128-bit identifier is generated for each MAP subtask using the UUID algorithm, recorded as the task ID, UUID (Universally Unique Identifier), that is, a universally unique identifier, also known as a globally unique identifier (GUID, Globally Unique Identifier). It is a software construction standard used to generate unique identifiers that are almost never repeated in distributed systems and concurrent environments; the MAP subtask executes predefined data preprocessing logic on the input ciphertext data shards, including filtering, conversion, feature extraction and other operations, to generate intermediate result data;

[0013] Process the intermediate result data of all generated MAP subtasks and construct the Reduce stage subtask; traverse the intermediate result data of the MAP subtask, read each data in turn, extract the keywords in the data, group the data according to the hash value of the keyword, and divide the data with the same hash value into the same group; traverse the grouped data, create a Reduce subtask for each data group, and use the data of the group as the input of the Reduce subtask; use the UUID algorithm to generate a unique 128-bit identifier for each Reduce subtask, recorded as the task ID; d. The Reduce subtask performs calculation logic such as merging and aggregation on multiple data in the same input group to generate one or more ciphertext output result data.

[0014] According to the input and output data relationship between MAP subtasks and Reduce subtasks, a subtask dependency graph is constructed; each MAP subtask and each Reduce subtask is used as a node of the dependency graph, and the attributes of the node include: task ID, task type, input shard ID, and output group ID; for each MAP subtask, find out the downstream Reduce subtask corresponding to its output data, and add a directed edge from the MAP subtask node to the Reduce subtask node to indicate that the output data of the MAP subtask is the input of the Reduce subtask; according to the constructed subtask dependency graph, a DAG graph is constructed. Check whether there is an MPC relationship between the Reduce subtask nodes, that is, multiple Reduce subtasks need to perform privacy-preserving joint calculations on the output results of different MAP subtasks; if so, merge multiple Reduce nodes with MPC relationships into an MPC super node; traverse each directed edge in the dependency graph, determine the starting and ending points of the directed edge according to the associated task ID, and construct a DAG graph represented by an adjacency list; use the Kahn topological sorting algorithm to detect loops in the DAG graph, and throw an exception if a loop is found; after confirming that the DAG graph has no loops, use it as the final dependency relationship of the subtask to be processed.

[0015] Among them, in the Map Reduce programming model, MAP subtask and Reduce subtask are two key task types, which complete different stages in the data processing flow. MAP subtask is the first processing stage in the Map Reduce framework. In this stage, the MAP subtask preprocesses the input data, such as filtering, conversion, feature extraction and other operations, and outputs the processed intermediate result data. Specifically, the MAP subtask receives a key-value pair as input, executes the predefined processing logic on the key-value pair, and outputs the processed intermediate result data in the form of a key-value pair. A MAP subtask can process one or more input key-value pairs and generate zero, one or more output key-value pairs. MAP subtasks are usually executed in parallel, that is, multiple MAP subtasks can process different input data at the same time to improve the execution efficiency of the entire task. Reduce subtask is the second processing stage in the Map Reduce framework. In this stage, the Reduce subtask aggregates, reduces and other operations on the intermediate result data output by the MAP stage, and outputs the final result. Specifically, the Reduce subtask receives the key-value pairs output by the MAP stage as input, where key-value pairs with the same key are grouped and passed to the same Reduce subtask. The Reduce subtask executes predefined aggregation logic on each grouped key-value pair, such as sum, average, maximum, etc., and outputs the aggregated results in the form of key-value pairs. The number of Reduce subtasks is usually less than that of MAP subtasks, because the Reduce stage mainly performs reduction and aggregation operations on the output results of the MAP stage. Key-value pairs in the same group will only be assigned to one Reduce subtask for processing to ensure the correctness of the results.

[0016] Furthermore, in the consensus mechanism of each shard chain, VDF is combined to form a hybrid consensus mechanism, including: in the consensus mechanism of each shard chain, the verifiable delay function VDF algorithm is used as one of the required consensus algorithms, and the timing proof of the data on the shard chain is provided based on the VDF algorithm to prevent the historical data of the shard chain from being tampered with; the computing power index of the current shard chain is obtained, and the index value is compared with the preset capacity threshold, and the security level of the shard chain is determined according to the comparison result; if the security level of the shard chain is medium or below, at least two consensus algorithms are randomly selected from the proof of work PoW, proof of stake PoS, delegated equity proof DPoS and Byzantine fault-tolerant PBFT consensus algorithm as candidate block confirmation algorithms; if the security level of the shard chain is medium In the above, at least one consensus algorithm is randomly selected from PoW, PoS, DPoS and PBFT consensus algorithms as a candidate block confirmation algorithm; the VDF timing proof algorithm is combined with the candidate block confirmation algorithm to form a hybrid consensus mechanism for the current shard chain; during the block synchronization process, the hybrid consensus mechanism performs the following operations on the block data: the block header data of the block to be synchronized is used as the VDF input, and the VDF calculation is performed to obtain the timing proof of the block header; the complete data of the block to be synchronized is used as the input of the block confirmation algorithm, and the block confirmation calculation is performed to obtain the confirmation proof of the block; the timing proof of the block header and the confirmation proof of the block are packaged into the block data structure to form a complete block with timing proof and confirmation proof.

[0017] The hybrid consensus mechanism is deployed to the consensus node corresponding to the current shard chain as the consensus algorithm of the shard chain; during the blockchain synchronization process, the consensus node implements the following processing flow by executing the hybrid consensus mechanism: the consensus node broadcasts the newly generated block data to other consensus nodes, and starts to execute the hybrid consensus calculation at the same time to obtain the timing proof and confirmation proof of the block; after receiving the block data, other consensus nodes independently execute the hybrid consensus calculation to obtain the timing proof and confirmation proof of the block; each consensus node compares the calculated block timing proof and confirmation proof, and if they are consistent, the block is appended to the local shard chain; otherwise, exception handling is performed; the consensus node synchronizes the hybrid consensus execution result to the shard chain, updates the global state of the shard chain, and completes a block synchronization cycle; repeat the processing flow, and in the process of continuous synchronization of blocks in the shard chain, the hybrid consensus mechanism continuously attaches timing proofs and confirmation proofs to new blocks to ensure that the timing of the shard chain cannot be tampered with and the security of block confirmation.

[0018] Furthermore, for subtasks with data dependencies, after the upstream subtask completes the calculation, the calculation result is sent to the shard chain where the downstream subtask is located through the cross-chain communication protocol, including: obtaining the predecessor subtask set of the current subtask from the DAG, and listening to the smart contract events of the shard chain where the predecessor subtask is located; after the upstream subtask is calculated, the shard chain where it is located calls the cross-chain communication contract and writes the calculation result to the specified storage location of the shard chain where the downstream subtask is located; after the cross-chain writing is completed, the cross-chain communication contract of the shard chain where the upstream subtask is located triggers the smart contract event and notifies the shard chain where the downstream subtask is located; after the shard chain where the downstream subtask is located listens to the event, it accesses the specified storage location to read the calculation result of the upstream subtask, uses it as input to execute the local smart contract, and performs the calculation of the current subtask.

[0019] Furthermore, multi-party secure computing (MPC) is adopted to obtain the global calculation results of data processing without leaking the original data of each data owner, including: decomposing the data processing tasks assigned to each shard chain into multiple subtasks, and assigning the subtasks to multiple data owners; wherein, the input data of the subtask is obtained through the DHT mapping table, and the output result is written back to the DHT after secret sharing; each data owner adopts the secret sharing protocol MMS to perform secret sharing processing on the homomorphically encrypted input data; wherein, the shared result is transmitted between data owners through DHT for local subtask calculation; each data owner only executes the assigned subtask locally, and writes the shared intermediate result back to the DHT for MPC global calculation to read; the MPC protocol is adopted to complete the global calculation of all subtasks, and the generated local sharing is used as input, and the global ciphertext result of the MPC calculation is written back to the main chain through cross-chain communication, and the main chain restores the plaintext result through homomorphic decryption and triggers a notification event.

[0020] Furthermore, each data owner only executes the assigned subtask locally, including: reading the input data shard corresponding to the subtask assigned to this node from the DHT mapping table; using the secret sharing protocol MMS to reconstruct the input password of the subtask, complete local calculations, and apply the MMS protocol to the calculation results to generate a shared share of the intermediate results; writing the shared intermediate results back to the corresponding position in the DHT mapping table, and the shared results flow between data owners via the DHT, and finally converge to the MPC global calculation.

[0021] Furthermore, the MPC multi-party secure computing protocol is adopted to obtain the global computing results of the data processing tasks without leaking the original data of each data owner, including: using the generated local MMS shared results of each subtask as the input secret of MPC, and completing the global aggregation of subtask results through MPC; wherein, the input of MPC comes from the sharing of intermediate results distributed on DHT; using the blind computing method for MPC aggregation, each node encrypts the local shared results based on a random mask, and removes the mask after multiple rounds of interactive computing to obtain the ciphertext form of the global result; using cross-chain communication to write the global ciphertext results calculated by MPC back to the main chain, and triggering the result decryption and verification smart contract on the main chain; each data node performs a threshold signature on the hash value of the ciphertext result; the main chain verifies the correctness and integrity of the MPC execution according to the threshold signature of the global result, uses homomorphic decryption to restore the final result plaintext, and notifies the plaintext result to each data node through an event.

[0022] Specifically, blind computing is used to achieve secure aggregation computing of multi-party data without leaking the original data of each party, including: Each node generates a random mask: Each node participating in the MPC calculation locally generates a random number r i , as the mask of local data. The selection of random numbers must meet certain conditions, such as the sum of random numbers is 0, that is, ∑r i = 0. Each node uses a random mask r i For local intermediate result data d i Perform mask processing to obtain the masked data m i =d i +r i Each node will mask the data m i Submit to the MPC network for calculation without submitting the original data i The MPC network collects the mask data m submitted by all nodes. i , and perform multiple rounds of interactive calculations, such as secure addition, secure multiplication, etc. Suppose the final calculation result is M=f(m 1 ,m 2 ,......,m n ), where f represents the secure computation logic of MPC. According to the properties of random mask, the computation result M of masked data and the computation result D of original data are equal to f(d 1 ,d 2 ,......,d n ) has the following relationship

[0023] M=f(m 1 ,m 2 ,......,m n )=f(d 1 +r1 ,d 2 +r 2 ,......,d n +r n )=f(d 1 ,d 2 ,......,d n )+f(r 1 ,r 2 ,......,r n ), = D + ∑ r i

[0024] Since ∑r i , so M = D, that is, the calculation result of the masked data is equal to the calculation result of the original data. Therefore, by subtracting the sum of the random mask ∑r from M i , the calculation result D of the original data can be restored. The MPC network uses the restored calculation result D as the final global aggregation result and writes it back to the main chain or outputs it to each node.

[0025] Compared with the prior art, the advantages of this application are:

[0026] This solution builds a heterogeneous multi-chain platform composed of multiple shard chains. For big data processing tasks, the system first builds a DAG dependency graph composed of multiple subtasks, and then dynamically maps subtasks to different shard chains based on task complexity and the computing power of each shard chain to achieve parallel processing of tasks. Since each shard chain runs independently and has its own consensus mechanism and computing resources, multiple shard chains execute different subtasks at the same time, which greatly improves the efficiency of task execution compared to traditional centralized serial processing.

[0027] In the consensus mechanism of each shard chain, this solution uses the verifiable delay function VDF as one of the consensus algorithms. VDF has the characteristic that the problem-solving time is proportional to the computing power, which can limit the block generation speed and suppress computing power attacks. At the same time, one of the consensus mechanisms such as PoW, PoS, DPoS, PBFT is selected to be combined with VDF to form a heterogeneous hybrid consensus. On the one hand, the consensus result is verified by VDF to ensure the verifiability of the block; on the other hand, the advantages of each consensus algorithm are fully utilized, which not only prevents the monopoly of computing power, but also takes into account the efficiency of block generation.

[0028] This solution adopts the secure multi-party computing MPC protocol, and multiple data owners collaborate to complete the subtask calculation. First, each party uses the secret sharing protocol MMS to split the local data and only upload the data fragments without leaking the original data. In the task execution phase, each party only executes its assigned subtask locally based on the secret shared data fragments. The resulting intermediate result fragments are securely transmitted between the parties through the DHT table for use by downstream subtasks. Finally, the blinding and interactive calculation of MPC are used to summarize the intermediate results of all parties. After multiple rounds of interaction, the random mask is removed to obtain the ciphertext form of the global result, and then the main chain is used for homomorphic decryption, which not only avoids privacy leakage but also completes the global calculation.

[0029] Before task execution, this solution performs fine-grained decomposition of data processing tasks and constructs a DAG graph consisting of multiple MAP and Reduce subtasks. The system monitors the load of each shard chain in real time, comprehensively considers factors such as DAG dependencies, data distribution locations, and network topology distances, flexibly schedules tasks based on subtasks, balances the load of each shard chain, and reduces data transmission between nodes, thereby reducing task execution latency and improving overall throughput.

[0030] For subtasks with data dependencies, after the upstream task is calculated, the intermediate results are sent to the shard chain where the downstream task is located through the cross-chain communication protocol. Specifically, the upstream task calls the cross-chain contract, writes the results to the specified location of the downstream shard chain, and triggers an event notification; after the downstream shard chain listens to the event, it reads the corresponding data and uses it as input to execute subsequent tasks. The cross-chain communication mechanism ensures the consistency of data between shard chains and provides strong support for parallel computing. At the same time, asynchronous communication is adopted between shard chains to further improve concurrency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0032] Figure 1 is an exemplary flow chart of a data management method for big data services according to some embodiments of the present application;

[0033] Figure 2 is an exemplary flow chart for obtaining ciphertext data according to some embodiments of the present application;

[0034] Figure 3 is an exemplary flow chart of distributed storage of ciphertext data according to some embodiments of the present application;

[0035] Figure 4This is an exemplary flowchart for obtaining the optimal shard chain according to some embodiments of the present application. DETAILED DESCRIPTION

[0036] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0037] Figure 1 It is an exemplary flow chart of a data management method for big data services according to some embodiments of the present application, including: collecting data to be processed; recording the process of collecting data in the form of transactions on the blockchain, and using zero-knowledge proof to verify the authenticity of the data under the premise of protecting privacy; homomorphically encrypting the data after authenticity verification, and then slicing the encrypted data using data slicing, and storing the sliced ​​data in multiple storage nodes in a dispersed manner; setting up a heterogeneous multi-chain platform, and allocating the processing tasks of the stored data to different shard chains in the heterogeneous multi-chain platform for parallel execution; at the same time, deploying a verifiable delay function VDF in each shard chain as one of the consensus algorithms of the shard chain, and according to the different shard chains of each shard chain, According to business needs, while choosing the VDF consensus algorithm, other consensus algorithms besides the VDF algorithm can be selected; the business logic of each shard chain is deployed on the corresponding shard chain in the form of a smart contract, and the cross-chain communication protocol is used for data interaction between cross-shard chains; multi-party secure computing MPC is used to decompose the data processing tasks assigned to each distribution chain into multiple subtasks, and the subtasks are assigned to multiple data owners; each data owner first secretly shares their own input data after homomorphic encryption, and then only executes the assigned subtask locally; finally, the calculation process of all subtasks is completed through the MPC protocol, and the global calculation results of data processing are obtained without leaking the original data of each data owner.

[0038] Figure 2 This is an exemplary flowchart for obtaining ciphertext data, data collection and chain records according to some embodiments of the present application. The system first collects the original data to be processed, such as environmental monitoring data, user behavior data, etc. For each batch of collected data, a data collection transaction is generated, and the transaction contains the following fields: Timestamp: record the time of data collection; Geographic location: record the location of data collection, such as GPS coordinates; Operator ID: record the identity information of the person who performs the data collection operation; Data hash: calculate the hash value of the collected original data as a data fingerprint; Submit the data collection transaction to the blockchain network, which is verified by the consensus node to ensure that the transaction is legal and valid, and package the transaction into a block to realize the storage of the transaction on the chain. On-chain transactions can provide traceable and tamper-proof audit evidence of the data collection process, but do not contain original data to protect data privacy.

[0039] Data authenticity proof generation: In this step, the data provider uses the zero-knowledge proof protocol to prove that it holds the real data corresponding to the hash value on the chain without providing the original data. The specific implementation method is as follows: construct a zero-knowledge proof circuit. The data provider constructs a zero-knowledge proof circuit locally to verify the authenticity of the original data. The input of the circuit includes: original data: the original data collected by the data provider; on-chain transaction hash: the hash value of the data collection transaction recorded on the blockchain corresponding to the original data; data collection metadata: including information such as the time, location, and operator of data collection; data collection requirements: predefined data collection specifications, such as data format, sampling frequency, etc.

[0040] The logic function of the circuit is to verify two conditions: whether the original data is consistent with the hash value in the on-chain transaction; whether the data collection metadata meets the predefined collection requirements. The output of the circuit is a binary value: if and only if both verification conditions are met, the circuit outputs 1; otherwise, the circuit outputs 0. Generate a zero-knowledge proof. The data provider uses a zero-knowledge proof algorithm such as zk-SNARK to generate a proof for the constructed circuit. The content of the proof is: without providing the circuit input (i.e., the original data), there is a set of inputs that satisfy the circuit logic, so that the output of the circuit is 1. The proof generation process requires the use of the data provider's private key to ensure the unforgeability of the proof, that is, only the data provider who actually holds the original data and the corresponding private key can generate a valid proof. Submit the proof to the chain. The data provider submits the generated zero-knowledge proof to the blockchain network and stores it in association with the corresponding data collection transaction. In this way, the data authenticity proof is closely associated with the data collection process information, which is convenient for subsequent verification.

[0041] Data authenticity verification: In this step, the data user verifies the data authenticity certificate submitted by the data provider to ensure the authenticity of the data. The specific implementation method is as follows: To verify the data collection transaction, the data user first obtains the data collection transaction related to the target data from the blockchain, and verifies the integrity and legality of the transaction, mainly including: verifying the correctness of the transaction hash to ensure that the transaction has not been tampered with; verifying the integrity and format legality of the data hash, collection metadata and other fields contained in the transaction; verifying that the transaction has been consensused and confirmed by the blockchain network to prevent forged transactions; this step ensures the authenticity and non-tamperability of the information in the data collection process.

[0042] To verify the authenticity of the data, the data user obtains the zero-knowledge proof generated by the data provider for the target data, and uses zero-knowledge proof verification algorithms such as zk-SNARK to verify the validity of the proof. The verification process takes the following information as input: the zero-knowledge proof submitted by the data provider; the data collection transaction hash corresponding to the target data obtained from the blockchain; the principle of the verification algorithm is: according to the characteristics of zero-knowledge proof, if the proof is valid, it means that the data provider does know a set of inputs that satisfy the circuit logic, that is, the original data held by the data provider is consistent with the transaction hash value on the chain, and the data collection process meets the predefined requirements. The entire verification process does not require the data user to access or obtain the original data, thereby protecting the confidentiality of the data.

[0043] Confirm the authenticity of the data. The data user makes a final judgment based on the data collection transaction verification results and the data authenticity proof verification results. Only when the transaction verification and the proof verification are passed, the data user is sure that the original data held by the data provider corresponds to the hash value on the chain, and the collection process is fully compliant, and the data is considered authentic and reliable; otherwise, if the transaction verification fails or the proof verification fails, the data user considers the data unreliable and refuses to use the data. Data encrypted transmission. After the data authenticity verification is passed, the data provider uses a fully homomorphic encryption algorithm such as BGV to encrypt the original data locally to obtain the ciphertext data. Then, the data provider transmits the encrypted ciphertext data to the data user. Due to the use of fully homomorphic encryption, the data user can directly perform calculations on the ciphertext data without decryption, which not only protects data privacy but also makes data available.

[0044] Homomorphic encryption: In this step, the original data that has passed the authenticity verification is homomorphically encrypted. The specific implementation method is as follows: Select a homomorphic encryption algorithm, and select a fully homomorphic encryption algorithm suitable for the target application scenario, such as the BGV (Brakerski Gentry Vaikuntanathan) algorithm. The BGV algorithm supports any number of additions and a finite number of multiplications on encrypted data, and is suitable for secure computing of privacy-preserving data. Generate a public-private key pair, and use the key generation function KeyGen() of the BGV algorithm to generate a pair of homomorphically encrypted public-private keys (p k ,s k ):

[0045] (p k ,s k )=KeyGen(λ,L,m), where λ is the security parameter, L is the number of multiplication layers, and m is the plaintext space. Encrypt the original data and perform homomorphic encryption on the original data m that has passed the authenticity verification. Use BGV's encryption function Enc(), with the public key pk as the parameter, to encrypt the plaintext m into the ciphertext c: c=Enc(p k,m). The encryption process embeds the plaintext m into a polynomial by randomly selecting noise, and then encrypts the coefficients of the polynomial to obtain the ciphertext c. The ciphertext is regarded as a secret, and the encrypted ciphertext c is regarded as a secret S for the subsequent secret sharing process.

[0046] Figure 3 This is an exemplary flowchart of distributed storage of ciphertext data shown in some embodiments of the present application, and a secret sharing polynomial is constructed. In this step, we construct a t-1 degree polynomial according to the secret sharing threshold t and the total number of shards n, use the ciphertext obtained by homomorphic encryption as the constant term of the polynomial, and randomly select other coefficients to form a secret sharing polynomial. The detailed implementation process is as follows: Determine the secret sharing parameters, select the secret sharing threshold t, which means that at least t shards need to be collected to restore the original secret. The selection of the threshold t requires a balance between security and availability, and usually satisfies 1≤t≤n. Select the total number of shards n, which indicates the number of copies of the original secret. The size of n determines the degree of dispersion of secret sharing. The larger n is, the less secret information each shard contains. Select a finite field GF(p), select a suitable large prime number p, and form a finite field GF(p). The coefficients and calculations of the secret sharing polynomial are all performed on this finite field. The selection of the prime number p needs to meet the following conditions: p should be large enough to ensure the security of operations on the finite field. p should be compatible with the secret sharing threshold t and the total number of shards n to avoid problems such as polynomial coefficient overflow.

[0047] Construct a secret sharing polynomial and randomly select a t-1 degree polynomial f(x) over the finite field GF(p): f(x) = a 0 +a 1 x+a 2 x 2 +......+a t-1 x t-1 modp; the ciphertext c obtained by homomorphic encryption is used as the constant term a of the polynomial 0 = S. Randomly select the other coefficients a of the polynomial 1 to a t-1 , the selection of coefficients should be uniformly random to ensure that the generated polynomial has good randomness. Through the above steps, we have constructed a t-1 degree polynomial f(x) with the homomorphic encryption ciphertext as the constant term and other coefficients randomly selected. This polynomial will serve as the basis for secret sharing and is used to generate data shards.

[0048] Generate data shards. After constructing the secret sharing polynomial f(x), we generate n data shards based on the polynomial. Each shard contains an x ​​coordinate value and a corresponding y coordinate value. The specific implementation process is as follows: select different x coordinate values, select n different x coordinate values ​​x i(i=1,2,.......,n) as the identifier of each shard. The selection of x_i should satisfy the following conditions: i should be an element of the finite field GF(p). i There should be no obvious pattern between them to prevent attackers from using the x i The value can be used to infer the location of other shards. i Think of it as the shard number, and the number should be unique when selected.

[0049] Calculate the y-coordinate value of the slice, for each selected x-coordinate value x i , substitute it into the secret sharing polynomial f(x) and calculate the corresponding y coordinate value y i :y i =f(x i )=a 0 +a 1 x i +a 2 x i 2 +......+a t-1 x i t-1 modp, the calculation process is performed on the finite field GF(p), ensuring that y_i is also an element in the finite field. Generate data slices and convert each coordinate pair (x i ,y i ) as a data shard, generating a total of n shards. Each shard is a part of the original secret, and a single shard will not leak the complete secret information. The form of data sharding can be expressed as: (x 1 ,y 1 ),(x 2 ,y 2 ),......,(x n ,y n ).

[0050] Generate private key shards. In this step, we use the threshold signature scheme (such as Shamir secret sharing) to generate private key shards, split the private key into multiple shards, and distribute them to different signing participants. The specific implementation process is as follows: Select signing participants, select a part of multiple storage nodes as signing participants, and participate in the generation of public-private key pairs and the sharding of private keys. The selection of signing participants should consider the following factors: The credibility of the participants: Select relatively trustworthy nodes as signing participants to reduce the risk of private key leakage. The number of participants: Select an appropriate number of signing participants, which should not only meet the threshold requirements of the threshold signature scheme, but also should not be too many to avoid increasing communication and computing overhead. Diversity of participants: Select nodes of different types and geographical locations as signing participants as much as possible to improve the system's fault tolerance and anti-attack ability.

[0051] Generate a public-private key pair. The signing parties jointly generate a public-private key pair for a threshold signature scheme through a secure multi-party computing protocol (p k ,s k ). Common threshold signature schemes include Shamir secret sharing, Blakley secret sharing, Pedersen distributed key generation, etc. In the multi-party computing process, each participant contributes his own random number to jointly generate a public-private key pair, ensuring that the generation of the private key is distributed and no single participant can master the complete private key alone. The generated public key pk is open to all participants for subsequent signature verification.

[0052] Private key sharding, using the threshold signature scheme, the generated private key s k Split into n private key shards s ki (i=1,2,.......,n). Taking Shamir secret sharing as an example, the process of private key sharding is as follows: On the finite field GF(p), select a t-1 degree polynomial f(x), where t is a threshold value, satisfying 1<=t<=n. The private key sk is used as the constant term of the polynomial, that is, f(0)=s k . Randomly select the other coefficients of the polynomial to generate the complete polynomial f(x). Assign a different x coordinate value x to each signing party. i , calculate the corresponding y coordinate value y i =f(x i ). The coordinate pair (x i ,y i ) as the i-th private key shard ki , distributed to the i-th signing party. Each signing party only holds its own private key shard, does not know the shards of other parties, and cannot recover the complete private key alone. Through the above steps, we use the threshold signature scheme to k Split into n private key shards and distribute them to the selected signing parties. This private key sharding method greatly reduces the risk of private key leakage and improves the security of the system. At the same time, due to the use of threshold signatures, only when a sufficient number of signing parties (greater than or equal to the threshold value) cooperate can the complete private key be restored and signed, preventing single point failure and private key abuse.

[0053] Distributed storage, after completing data sharding and private key sharding, we will disperse these shards to multiple storage nodes to achieve distributed storage. The specific implementation process is as follows: randomly assign data shards, and generate n data shards (x i ,y i) is randomly distributed to n storage nodes, each node stores a data shard. The distribution process should be as random as possible to avoid regularly allocating shards to specific nodes and reduce the risk of data leakage. You can use methods such as random number generators or hash functions to randomly select storage nodes based on the shard number or hash value. Distribute private key shards and distribute the generated n private key shards to n nodes. ki Distribute to the corresponding storage nodes, each node holds a private key shard. The distribution of private key shards should be carried out through a secure communication channel to ensure that the shards will not be intercepted or tampered with during transmission. Encrypted communication protocols (such as SSL / TLS) or end-to-end encryption can be used to protect the confidentiality and integrity of private key shards.

[0054] Establish the correspondence between shards and nodes, record the storage nodes where each data shard and private key shard is located, and establish the correspondence between shards and nodes. This correspondence table can be stored on a trusted centralized server, or maintained in a distributed manner (such as blockchain) to ensure the reliability and consistency of the correspondence. The role of the correspondence table is to facilitate subsequent data retrieval and recovery. When you need to access a data shard or private key shard, you can quickly locate the node storing the shard based on the correspondence table.

[0055] Data recovery, in this step, the data user collects a certain number of data shards from the storage nodes, reconstructs the secret sharing polynomial using the Lagrange interpolation formula, and calculates the homomorphically encrypted ciphertext. The specific implementation process is as follows: Select enough data shards, and the data user randomly selects t nodes from n storage nodes, where t is the threshold of secret sharing, satisfying 1≤t≤n. The selected t nodes should be available and trustworthy to ensure that they can provide the correct data shards.

[0056] Collect data shards, and the data user sends data shard requests to the selected t nodes to obtain the data shards they hold (x i ,y i ), where i = 1, 2, ..., t. The transmission of data slices should be carried out through a secure communication channel to prevent the slices from being eavesdropped or tampered with during the transmission process. Reconstruct the secret sharing polynomial and use the collected t data slices (x i ,y i ), the data user reconstructs the secret sharing polynomial f(x) through the Lagrange interpolation formula: The Lagrange interpolation formula is based on the known t points (x i ,y i), solve a t-1 degree polynomial f(x) so that the polynomial passes through all known points. The interpolation calculation is performed on the finite field GF(p) to ensure the correctness of the calculation result. Calculate the homomorphically encrypted ciphertext and evaluate the reconstructed secret sharing polynomial f(x) at x=0, that is, calculate f(0). According to the principle of secret sharing, f(0) is the secret value S of the secret sharing, which is the homomorphically encrypted ciphertext c.

[0057] Data decryption and signature verification. After successfully recovering the homomorphically encrypted ciphertext, the data user needs to decrypt the ciphertext and verify the integrity and source credibility of the data. This process involves threshold signatures and homomorphically encrypted private key decryption. The specific implementation process is as follows: Select signature participants. The data user selects t nodes as signature participants from the storage nodes holding the data shards, where t is the threshold of the threshold signature. The selected signature participants should be consistent with the nodes selected during data recovery to ensure that they hold the correct private key shards. Generate a partial signature, and the data user sends the recovered ciphertext c to the selected t signature participants. Each participant uses the private key shard s it holds. ki Sign the ciphertext c and generate a partial signature σ i The generation process of some signatures depends on the specific threshold signature scheme used, such as Shamir threshold signature, BLS threshold signature, etc.

[0058] Aggregate signature: the data user collects partial signatures σ from t signing parties i (i=1,2,.......,t). According to the aggregation rule of the threshold signature scheme, t partial signatures are aggregated into a complete signature σ. The aggregated signature σ represents the joint authorization of all nodes participating in the signature, proving that the ciphertext c has been recognized by enough nodes. To verify the signature, the data user uses the public key p of the threshold signature k , verify the aggregate signature σ. The signature verification process depends on the specific threshold signature scheme, which usually includes verifying the correctness and integrity of the signature. If the signature verification passes, it means that the recovered ciphertext c is credible and comes from an authorized storage node. Decrypt the data. After the signature verification passes, the data user uses the homomorphic encryption private key s k Decrypt the ciphertext c. The decryption process is performed according to the definition of the homomorphic encryption algorithm, which is usually a mathematical operation based on the private key. After decryption, the plaintext data m is obtained, that is, the original shared data.

[0059] Figure 4It is an exemplary flowchart for obtaining the optimal shard chain shown in some embodiments of the present application, setting a heterogeneous multi-chain platform, and assigning the processing tasks of the stored data to different shard chains in the heterogeneous multi-chain platform for parallel execution. Construct a directed acyclic graph DAG of the subtasks to be processed, divide the ciphertext data to be processed into multiple subtasks, and construct a directed acyclic graph DAG according to the dependencies between the subtasks. Data sharding, divide the ciphertext data to be processed into multiple data shards, and divide the ciphertext data into several data shards according to the size of the data and processing requirements. Each data shard contains a part of the ciphertext data, and the amount of data is relatively small, which is convenient for parallel processing. The size of the data shard can be adjusted according to the actual situation to balance the task granularity and parallel efficiency. Generate a unique identifier for the data shard, and use a hash function to generate a hash value for the content of each data shard. The hash value is used as the unique identifier of the corresponding data shard, recorded as the shard ID. Commonly used hash functions such as SHA-256 can generate a 256-bit hash value and have good anti-collision performance.

[0060] Construct MAP stage subtasks. Construct MAP stage subtasks based on data shards, and use each data shard as the input of a MAP subtask. Each MAP subtask is responsible for processing a data shard and preprocessing the ciphertext data therein. Assign a unique task ID to the MAP subtask. Assign a unique task ID to each MAP subtask to identify and track the task. The task ID can be generated by combining the data shard ID and an increasing sequence number to ensure uniqueness. The MAP subtask preprocesses the ciphertext data. The MAP subtask reads the assigned data shards and preprocesses the ciphertext data therein. Preprocessing can include data decryption, format conversion, feature extraction and other operations, depending on the specific application scenario. The MAP subtask outputs the preprocessed intermediate result data and marks the corresponding data shard ID. Construct Reduce stage subtasks. Process the intermediate result data of the MAP stage and collect the intermediate result data generated by all MAP subtasks. Construct Reduce stage subtasks according to the characteristics and processing requirements of the intermediate result data. The Reduce subtask is responsible for aggregating, calculating, filtering and other operations on the intermediate result data. 3.2 Determine the input data of the Reduce subtask, analyze the output data of the MAP subtask, and determine which data needs to be used as the input of the Reduce subtask. According to the relevance of the data and the calculation logic, combine the related intermediate result data into the input data set of the Reduce subtask.

[0061] Construct a subtask dependency graph, analyze the input-output relationship between MAP and Reduce subtasks, review the output data of MAP subtasks, and carefully check the output data of each MAP subtask to understand its data format, data volume, and data content. The output data of MAP subtasks is usually the intermediate result data after preprocessing, which includes the results of conversion and extraction of the original ciphertext data. The output data can be a structured key-value pair, vector representation, or other format, depending on the purpose of preprocessing and the needs of subsequent tasks. Review the input data of Reduce subtasks, analyze the input data requirements of each Reduce subtask, and determine the required data type, format, and source. The input data of Reduce subtasks usually comes from the output data of MAP subtasks, and may require the output data of multiple MAP subtasks to be aggregated or combined. The selection and organization of input data depends on the calculation logic and target results of Reduce subtasks. Determine the corresponding relationship between the output of MAP subtasks and the input of Reduce subtasks, and compare and match the output data of MAP subtasks with the input data of Reduce subtasks. Identify which MAP subtask output data is the input data of specific Reduce subtasks, and establish the data dependency relationship between them. This correspondence can be one-to-one, one-to-many, or many-to-one, depending on the distribution of data and the requirements of computing. Record the data dependencies between subtasks, and record and store the data dependencies between the determined MAP subtasks and Reduce subtasks. Tables, matrices, or other data structures can be used to represent these dependencies so that the subtask dependency graph can be constructed later. The recorded content includes the identifiers of the subtasks, the identifiers of the input and output data, and the dependency directions between them.

[0062] Construct a subtask dependency graph, create a data structure for the subtask dependency graph, and select an appropriate data structure to represent the subtask dependency graph. Common choices include adjacency lists, adjacency matrices, etc. According to the number of subtasks and the complexity of the dependency relationships, select a data structure that can effectively store and query the dependency relationships. Initialize the data structure of the subtask dependency graph to prepare for adding nodes and edges. Add subtask nodes and create corresponding nodes in the subtask dependency graph according to the identifiers of the MAP and Reduce subtasks. Each node represents an independent subtask, including MAP tasks and Reduce tasks. The attributes of the node can include the identifier of the subtask, the task type, the identifiers of the input and output data, etc. Add dependency edges between subtasks and add directed edges to the subtask dependency graph according to the recorded data dependencies between subtasks. The starting point of the directed edge is the subtask node that outputs the data, and the end point is the subtask node that depends on the data. Each directed edge represents the flow direction and dependency relationship of the data. The output data of the starting task is the input data of the end task.

[0063] Construct DAG based on the subtask dependency graph. Based on the subtask dependency graph, construct a directed acyclic graph DAG. DAG inherits the structure of the subtask dependency graph, and each node corresponds to a subtask. Through algorithms such as topological sorting, ensure that there is no loop in the DAG and satisfy the task dependency relationship. Create shard chain instances. According to the number of nodes in the directed acyclic graph (DAG) of the subtask to be processed, create a corresponding number of shard chain instances and form them into a heterogeneous multi-chain platform. Determine the number of shard chains, analyze the number of subtask nodes in the DAG, traverse the DAG, and count the total number of subtask nodes contained therein. Subtask nodes include subtasks in the MAP phase and subtasks in the Reduce phase. The statistical results represent the total number of subtasks that need to be processed. Determine the number of shard chain instances to be created. In order to fully utilize the parallel processing capabilities of the heterogeneous multi-chain platform, the number of shard chain instances created should be at least equal to the number of subtasks. The number of shard chain instances can be appropriately increased according to the actual hardware resources and performance requirements. The selection of the number of shard chain instances requires balancing load balancing and resource utilization. 1. Create a heterogeneous multi-chain platform, create shard chain instances, and create a corresponding number of shard chain instances according to the determined number of shard chain instances. Each shard chain instance is an independent blockchain with its own consensus mechanism, data storage, and smart contract execution environment. Shard chain instances can run on different physical nodes or virtual machines to make full use of distributed computing resources. 2. Configure shard chain instances, and configure necessary parameters and settings for each shard chain instance, such as consensus algorithm, block size, block time, etc. 3. Ensure that the communication and data synchronization mechanisms between shard chain instances operate normally. 4. Assign unique identifiers to shard chain instances for subsequent task allocation and result collection. 5. Initialize shard chain instances, and deploy necessary smart contracts and related components on each shard chain instance. 6. Initialize the state and data storage of the shard chain instance to prepare for receiving and processing subtasks. 7. Start the shard chain instance to ensure that it operates normally and can communicate with other shard chain instances.

[0064] Record the mapping relationship between subtasks and shard chains, and use the distributed hash table (DHT) to record the mapping relationship between subtasks and shard chains, so that the corresponding shard chains can be quickly located when tasks are assigned and results are collected. Build a distributed hash table (DHT) and select a DHT implementation. Select a suitable DHT implementation based on the system scale, performance requirements, and available open source implementations. Common DHT implementations include Chord, Kademlia, Pastry, etc., which provide efficient key-value pair storage and query in a distributed environment. The selected DHT implementation should be compatible with the architecture and communication protocol of the heterogeneous multi-chain platform. Deploy DHT nodes and deploy DHT-implemented nodes on the nodes of the heterogeneous multi-chain platform. DHT nodes are responsible for storing and maintaining the mapping relationship between subtasks and shard chains. Ensure that the network connection between DHT nodes and shard chain instances is normal so that the mapping relationship can be updated and queried in a timely manner. 1.3 Configure DHT parameters. Configure DHT nodes according to the specific parameters of the DHT implementation. Configuration items may include replication factor, fault tolerance mechanism, routing table size, etc. to ensure the reliability and performance of DHT. Appropriately adjust the DHT parameter settings to adapt to the scale and load requirements of the heterogeneous multi-chain platform.

[0065] Generate subtask ID and shard chain ID. Generate subtask ID. For each subtask node in the DAG, generate a unique subtask ID. The subtask ID can be obtained by hashing the unique identifier of the subtask node. The hash operation can use a secure hash function, such as SHA-256, to ensure the uniqueness and uniform distribution of the subtask ID. Generate shard chain ID. For each shard chain instance, generate a unique shard chain ID. The shard chain ID can be obtained by hashing the unique identifier of the shard chain instance. Also use a secure hash function, such as SHA-256, to ensure the uniqueness and uniform distribution of the shard chain ID. Build a mapping table from task to shard chain, using the subtask ID as the key and the shard chain ID as the value. In the DHT, use the subtask ID as the key and the corresponding shard chain ID as the value. The key-value pair is in the form of: (subtask ID, shard chain ID). Through this mapping relationship, the shard chain responsible for processing the task can be quickly queried based on the subtask ID. Store the mapping relationship in the DHT and store the generated key-value pair in the DHT. DHT is responsible for distributing and storing key-value pairs on different DHT nodes, ensuring the distributed storage and high availability of mapping relationships. The routing and query mechanism of DHT ensures efficient access and update of mapping relationships.

[0066] Select the best shard chain for the subtask. According to the computing power information of the shard chain where the predecessor task node of the subtask is located, select the best shard chain for the current subtask to achieve load balancing and efficient execution. Obtain the predecessor task node set, traverse the DAG graph, start from the starting node of the DAG graph, and traverse the DAG graph in the order of topological sorting. For each subtask node, determine its predecessor task node set. Determine the predecessor task node, which is the input data source of the current subtask node. By analyzing the directed edges in the DAG graph, find all the predecessor task nodes pointing to the current subtask node. Add these predecessor task nodes to the predecessor task node set of the current subtask node. Obtain the computing power information of the shard chain where the predecessor task node is located, query the DHT mapping table, and query the mapping relationship between the predecessor task node and the shard chain in the DHT mapping table. Obtain the ID of the shard chain where the predecessor task node is located through the ID of the predecessor task node. Obtain the computing power information of the shard chain, and query the computing power information of the corresponding shard chain according to the shard chain ID. Computing power information can include indicators such as the hardware configuration, load status, network bandwidth, etc. of the shard chain. This information can be obtained in real time through the monitoring and measurement components on the shard chain.

[0067] Select the optimal shard chain, evaluate the computing power of the shard chain, and evaluate the performance and load status of each shard chain based on the obtained shard chain computing power information. A weighted scoring mechanism can be used to comprehensively consider factors such as the hardware configuration, current load, and network bandwidth of the shard chain. The scoring result reflects the computing power and availability of the shard chain. Select the optimal shard chain, and select the shard chain with the best computing power as the execution environment of the current subtask node based on the scoring result. The optimal shard chain should have high computing performance and low load to ensure efficient execution of subtasks. If the scores of multiple shard chains are similar, a load balancing strategy can be used to assign subtasks to different shard chains. Update the DHT mapping table and write the mapping relationship between the current subtask node and the selected optimal shard chain into the DHT mapping table. The updated mapping relationship is: (subtask ID, optimal shard chain ID).

[0068] By updating the DHT mapping table, ensure that subsequent task scheduling and data access can correctly locate the optimal shard chain. Convert subtask processing logic into smart contracts and deploy them. In this step, we convert the processing logic of each subtask into smart contract code and deploy it to the corresponding shard chain so that the subtask can be executed on the shard chain. Convert subtask processing logic, analyze the processing logic of subtasks, and analyze the processing logic of each subtask in detail to determine the input data, processing steps, and output results. The processing logic may include operations such as data conversion, filtering, aggregation, and encryption. Clarify the computational complexity and resource requirements of the subtask. Design the smart contract interface. According to the processing logic of the subtask, design the interface and function of the smart contract. The interface should clearly define the input parameters, return values, and function functions. Consider the reusability and modular design of the smart contract. Implement the smart contract code. Use an appropriate smart contract programming language, such as Solidity, to write the smart contract code. Convert the processing logic of the subtask into functions and algorithms in the smart contract. Ensure the correctness, security, and gas optimization of the smart contract code.

[0069] Determine the shard chain information, query the DHT mapping table, and query the corresponding shard chain ID in the DHT mapping table according to the subtask ID. The DHT mapping table stores the mapping relationship between subtasks and shard chains. Obtain the deployment information of the shard chain. According to the shard chain ID, obtain the deployment information of the corresponding shard chain. The deployment information includes the network address, consensus mechanism, smart contract operating environment, etc. of the shard chain. This information is necessary for the deployment and call of smart contracts. Deploy smart contracts, connect to the shard chain, and use the deployment information of the shard chain to establish a connection with the shard chain. The connection process may require identity authentication and authorization. Deploy smart contracts and deploy the converted smart contract code to the smart contract operating environment of the shard chain. The deployment process may require a certain handling fee or gas fee. Record the deployed smart contract address for subsequent calls and interactions. Query the storage location of the input data shard, analyze the input data dependency of the smart contract, and determine the required input data shard based on the input parameters of the smart contract. The input data shard may come from the output result of the predecessor task node. Query the DHT mapping table and query the storage location of the input data shard according to its ID. The DHT mapping table records the mapping relationship between data shards and shard chains. Get the shard chain information where the input data shard is located.

[0070] Initiate a cross-chain data read request, connect to the shard chain where the input data shard is located, and use the deployment information of the shard chain where the input data shard is located to establish a connection with the shard chain. Initiate a data read request. When the smart contract is executed, initiate a cross-chain data read request to obtain the required data from the shard chain where the input data shard is located. The cross-chain data read request should contain the necessary authentication and authorization information. The result of the data read request will be used as the input data of the smart contract. Process the data read result. After receiving the result of the cross-chain data read request, the smart contract will perform necessary parsing and processing on the data. Pass the read input data to the corresponding function in the smart contract for calculation.

[0071] Design a hybrid consensus mechanism for the shard chain. Design a hybrid consensus mechanism for each shard chain, combining a verifiable delay function (VDF) with other consensus algorithms to improve the security and efficiency of the consensus. Fix VDF as one of the consensus algorithms. Select a VDF algorithm. Select a VDF algorithm that is suitable for the blockchain environment, such as Pietrzak VDF or Wesolowski VDF. The VDF algorithm should be verifiable, resistant to attacks, and have a moderate computational delay. Determine the VDF parameters. Determine the parameter settings of the VDF algorithm based on the security requirements and performance requirements of the shard chain. Parameters can include computational difficulty, number of iterations, output length, etc. Integrate VDF into the consensus mechanism. Integrate the VDF algorithm into the consensus mechanism of the shard chain as one of the necessary consensus steps. The calculation result of VDF will be used as a time proof of the block to verify the validity and time sequence of the block. Select other consensus algorithms. Evaluate the computing power of the shard chain. Analyze factors such as the hardware configuration, network bandwidth, and number of nodes of the shard chain to evaluate its computing power. According to the level of computing power, select a suitable other consensus algorithm. Select other consensus algorithms, and select at least one consensus algorithm from Proof of Work (PoW), Proof of Stake (PoS), Delegated Proof of Stake (DPoS), and Practical Byzantine Fault Tolerance (PBFT). The selected consensus algorithm should match the computing power of the shard chain, and take into account factors such as security, decentralization, and performance. You can choose to use a combination of multiple consensus algorithms according to specific needs.

[0072] Form a hybrid consensus mechanism, design a hybrid consensus process, combine VDF with other selected consensus algorithms, and design the overall process of hybrid consensus. Determine the execution order and interaction mode of each consensus algorithm in the hybrid consensus. Design the roles and responsibilities of the consensus nodes, as well as the verification and confirmation methods of the consensus results. Implement the hybrid consensus protocol, and implement the corresponding consensus protocol and algorithm according to the designed hybrid consensus process. Develop the software module of the consensus node, including VDF calculation, execution of other consensus algorithms, broadcasting and verification of consensus results, etc. Ensure the correctness, security and efficiency of the hybrid consensus protocol. Deploy the hybrid consensus mechanism, select the consensus node, and select the appropriate node as the consensus node according to the node configuration and network topology of the shard chain. The consensus node is responsible for executing the hybrid consensus protocol, generating and verifying blocks. Deploy the hybrid consensus software, and deploy the implemented hybrid consensus software to the selected consensus node. Configure the network connection, security policy and other necessary parameters of the consensus node. Start the consensus node to join the consensus network of the shard chain.

[0073] Execute the hybrid consensus mechanism, the consensus node generates blocks, and the consensus node generates new blocks according to the hybrid consensus protocol. The block generation process involves the packaging of transactions, the acquisition of timestamps, the calculation of VDFs, and the execution of other consensus algorithms. Broadcast blocks and consensus calculation results, the consensus node broadcasts the newly generated blocks to other consensus nodes. At the same time, the consensus node performs hybrid consensus calculations to obtain the timing proof (VDF calculation results) and confirmation proof (calculation results of other consensus algorithms) of the block. The consensus node broadcasts the block and consensus calculation results to other consensus nodes for verification. Verify and confirm the block, after receiving the block and consensus calculation results, other consensus nodes verify them. The verification process includes checking the validity of the block, the correctness of the timing proof, and the validity of the confirmation proof. If the verification passes, the consensus node adds the block to the local shard chain and updates the global state of the shard chain. Complete block synchronization, the consensus node completes the synchronization of the new block through broadcasting and verification. The local shard chains of all consensus nodes contain the new block and reach consistency in the global state. The block synchronization process of the shard chain is completed and continues to the next round of consensus process.

[0074] Process the data dependencies between subtasks, process the data dependencies between subtasks, and ensure that the calculation results of upstream subtasks can be correctly passed to downstream subtasks to support continuous task execution. Get the predecessor subtask set, traverse the DAG, start from the starting node of the DAG, and traverse the DAG in the order of topological sorting. For each subtask node, determine its predecessor subtask node set. Determine the predecessor subtask, which is the input data source of the current subtask. By analyzing the directed edges in the DAG, find all the predecessor subtask nodes pointing to the current subtask node. Add these predecessor subtask nodes to the predecessor subtask set of the current subtask. Listen to the smart contract events of the shard chain where the predecessor subtask is located, obtain the shard chain information of the predecessor subtask, and query the DHT mapping table according to the ID of the predecessor subtask to obtain the shard chain information where it is located. The shard chain information includes the network address of the shard chain, the consensus mechanism, etc.

[0075] Connect to the shard chain where the predecessor subtask is located, and use the network address of the shard chain to establish a connection with the shard chain where the predecessor subtask is located. The connection process may require authentication and authorization. Listen to smart contract events and deploy a cross-chain communication contract on the shard chain where the predecessor subtask is located. The cross-chain communication contract defines events to notify the shard chain where the downstream subtask is located about the availability of the calculation results of the upstream subtask. The shard chain where the downstream subtask is located obtains the status of the calculation results of the upstream subtask by listening to these events. Write the calculation results across the chain, the upstream subtask completes the calculation, and the upstream subtask completes the calculation in the smart contract of the shard chain where it is located. The calculation results are used as the output data of the upstream subtask. Call the cross-chain communication contract, and the smart contract of the shard chain where the upstream subtask is located calls the cross-chain communication contract. The cross-chain communication contract provides the function of writing the calculation results to the shard chain where the downstream subtask is located. Write the calculation results, and the cross-chain communication contract writes the calculation results of the upstream subtask to the specified storage location of the shard chain where the downstream subtask is located. The writing process may involve cross-chain data transmission and consistency guarantee mechanisms.

[0076] Trigger smart contract event, confirm writing is complete, the cross-chain communication contract confirms that the calculation result has been successfully written to the shard chain where the downstream subtask is located. Trigger event, the cross-chain communication contract triggers a smart contract event, notifying the shard chain where the downstream subtask is located about the availability of the calculation result. The event contains information such as the ID of the upstream subtask and the storage location of the calculation result. Read the calculation result of the upstream subtask, listen to the event, the event listener of the shard chain where the downstream subtask is located captures the event triggered by the cross-chain communication contract. The event contains relevant information about the calculation result of the upstream subtask. Access the specified storage location, the shard chain where the downstream subtask is located accesses the storage location of the calculation result of the upstream subtask according to the storage location information provided in the event. The storage location can be the local storage of the shard chain where the downstream subtask is located, or it can be the storage on other shard chains. Read the calculation result, the shard chain where the downstream subtask is located reads the calculation result of the upstream subtask from the specified storage location. The reading process may involve cross-chain data transmission and consistency verification mechanism.

[0077] Execute the local smart contract and use the calculation result as input. The shard chain where the downstream subtask is located will read the calculation result of the upstream subtask as input data. The input data will be passed to the smart contract corresponding to the downstream subtask. Execute the smart contract. The shard chain where the downstream subtask is located executes the corresponding smart contract. The smart contract uses the calculation result of the upstream subtask as input to calculate the current subtask. The calculation process is carried out according to the specific logic and algorithm of the subtask. Generate the calculation result. After the smart contract completes the calculation, it generates the calculation result of the current subtask. The calculation result is the output data of the current subtask and can be used by subsequent subtasks.

[0078] Privacy-preserving computing: The system uses MPC to complete global computing without leaking the original data. Data processing tasks are decomposed into subtasks and assigned to data owners. Task decomposition: According to the data processing tasks assigned to each shard chain, they are decomposed into multiple subtasks. The division of subtasks should consider factors such as data distribution, computational complexity, and data dependencies. Each subtask should have clear input data and expected output results. Subtask allocation: Subtasks are assigned to multiple data owners, and each data owner is responsible for executing a part of the subtasks. The allocation of subtasks should be as balanced as possible to avoid excessive computational load on a single data owner. The selection of data owners can be based on factors such as their computing power, data relevance, and credibility. Obtain the input data of the subtask: The input data of the subtask is obtained through the DHT mapping table. The data owner queries and obtains the corresponding input data from the DHT mapping table according to the requirements of the subtask. The input data is stored in a distributed manner in the DHT to ensure data availability and fault tolerance.

[0079] Secret sharing output results: The output results of the subtask need to be written back to the DHT after secret sharing. The data owner applies the secret sharing protocol to the output results of the subtask to generate multiple shared shares. The number of shared shares should be determined based on security requirements and fault tolerance requirements. Write back the shared results: The data owner writes the generated shared shares back to the corresponding position of the DHT mapping table. The shared shares are stored in a distributed manner in the DHT to ensure the availability and fault tolerance of the results. Other data owners can obtain shared shares from the DHT for subsequent MPC global calculations.

[0080] The data owner uses a secret sharing protocol to process the input data, homomorphically encrypts the input data, and sends the original input data to S3 for homomorphic encryption. After receiving the original input data, the data is encrypted using the specified homomorphic encryption algorithm. Common homomorphic encryption algorithms include Paillier encryption and BGV encryption. Homomorphically encrypted data has a special property, that is, certain operations are performed on the encrypted data, and the results are consistent with the results of the same operations on the original data after decryption. This allows calculations to be performed on encrypted data without decrypting the data. The encrypted input data is returned to the data owner. The data owner now holds the homomorphically encrypted input data, not the original data. Homomorphic encryption ensures the confidentiality of the input data. Even if the data is intercepted or leaked during transmission, the attacker cannot recover the original data from the encrypted data, thereby preventing data leakage.

[0081] Secret sharing processing: Each data owner uses the secret sharing protocol MMS to process the homomorphically encrypted input data. The MMS protocol is a multi-party secure computing protocol used to share data between multiple participants and perform calculations without leaking the original data. The MMS protocol divides the encrypted data into multiple shared shares. Specifically, the MMS protocol divides the encrypted data into n shared shares and distributes these shared shares to n data owners. Each data owner only holds a portion of the shared shares. The generation of shared shares should ensure privacy and security. The MMS protocol uses mathematical techniques such as polynomial interpolation and threshold secret sharing to generate shared shares. These techniques ensure that a single data owner cannot recover the original data from the shared shares he holds, thereby protecting the privacy of the data. The generation of shared shares should also consider fault tolerance and recoverability. The MMS protocol usually adopts a threshold secret sharing scheme, which requires at least t shared shares to recover the original data. In this way, even if some data owners fail or are malicious, as long as there are a sufficient number of honest participants, the original data can be recovered, ensuring the recoverability of the calculation.

[0082] Sharing result delivery, sharing results are delivered between data owners through a distributed hash table (DHT). DHT is a decentralized key-value storage system that allows participating nodes to efficiently store and retrieve data. Each data owner writes the shared share it generates to the corresponding location of the DHT. Specifically, the data owner associates the shared share with a specific key according to predefined rules and stores the key-value pair in the DHT. Other data owners can obtain the required shared share from the DHT according to the same rules. The data owner can obtain the corresponding shared share by querying the value associated with a specific key. DHT provides an efficient data storage and retrieval mechanism with the characteristics of decentralization and fault tolerance. Even if some nodes fail, other nodes can still obtain the required shared share from the DHT, ensuring the availability of data.

[0083] Local subtask calculation: After the data owner obtains the shared shares related to it from the DHT, it uses these shared results to perform local subtask calculations. Each data owner can only access the shared shares held by itself and cannot recover the shared shares of other data owners. During the local calculation process, the data owner uses the shared shares as input and executes the predefined calculation logic. The calculation logic can be simple arithmetic operations, statistical analysis, or machine learning algorithms, etc., depending on the application scenario and task requirements. The MMS protocol ensures that during the local calculation process, the data owner cannot recover the original data from the shared shares held by itself. Each data owner can only see his own shared shares and cannot know the shared shares of other data owners. Through the MMS protocol, each data owner can complete local calculations without leaking the original data. The calculation process is carried out independently in the local environment of each data owner, and there is no need to directly exchange the original data, thereby protecting the confidentiality of the data.

[0084] The data owner executes the local subtask and writes back the intermediate results, reads the input data shards, and reads the corresponding input data shards from the DHT mapping table according to the assigned subtask. The DHT mapping table maintains the correspondence between subtasks and input data shards. Each subtask has clearly defined input data requirements, such as a specific data range, data type, or data format. Based on these requirements, the data owner obtains the input data shards related to the subtask from the DHT. The DHT mapping table uses a distributed storage method to store the input data shards on multiple nodes. The data owner locates and obtains the required input data shards by querying the DHT mapping table. 1.4 In order to protect the confidentiality and privacy of data, each data owner can only access the data shards directly related to the subtask assigned to it. The data owner cannot obtain the input data shards of other subtasks or other data owners.

[0085] To reconstruct the subtask input, the data owner uses the secret sharing protocol MMS to reconstruct the input password of the subtask. The MMS protocol allows the input required for the subtask to be restored without leaking the original data. The reconstruction process of the MMS protocol requires the participation of multiple data owners. Each data owner holds a partial share, and a single data owner cannot independently restore the input data. The data owner uses the input data shards obtained from the DHT as the input of the MMS protocol, and collaborates with other data owners to jointly reconstruct the complete input password of the subtask. The MMS protocol uses technologies such as threshold secret sharing to ensure that only a certain number of honest participants can successfully reconstruct the input password. In this way, even if some data owners fail or are malicious, the security of the reconstruction process can still be guaranteed. The reconstruction process is completed through multiple rounds of interaction and calculation. Each data owner contributes his own share, and finally obtains the complete input password of the subtask for subsequent local calculations.

[0086] Local computing: The data owner uses the reconstructed subtask input password to perform computing tasks in the local environment. The local computing process is strictly limited to the data owner's own computing resources to ensure the confidentiality of the data. Local computing tasks can be various data processing, analysis or mining algorithms, such as data screening, aggregation, statistical analysis, machine learning, etc. The specific computing tasks depend on the application scenario and business needs. The data owner uses the input password of the subtask to perform calculations according to the predefined calculation logic. During the calculation process, the data owner can only access the input data he holds and cannot obtain the data of other data owners. The calculation result is the intermediate result of the subtask, which indicates the partial calculation progress. The intermediate result can be an intermediate variable, statistic, model parameter, etc., which is used for subsequent global summary and analysis.

[0087] Generate intermediate result sharing. For the calculated intermediate results, the data owner applies the MMS protocol to generate shared shares. The generation of shared shares is intended to protect the privacy and security of the intermediate results. The data owner uses the intermediate results as the input of the MMS protocol and uses techniques such as threshold secret sharing to divide the intermediate results into multiple shared shares. Each shared share only contains part of the intermediate result information. The number of shared shares should be determined based on security requirements and fault tolerance requirements. Generally, the number of shared shares should be greater than or equal to the number of participants to ensure sufficient redundancy and fault tolerance. The generation process of shared shares should ensure privacy and security. A single data owner cannot recover the complete intermediate result from the shared shares he holds, thereby protecting the confidentiality of the intermediate results.

[0088] Write back the shared results. The data owner writes the generated shared shares of the intermediate results back to the corresponding position of the DHT mapping table. The DHT mapping table maintains the correspondence between the shared shares of the intermediate results and the subtasks. The shared results are stored in a distributed manner in the DHT to ensure the availability and fault tolerance of the results. The distributed nature of the DHT enables the shared results to be stored in multiple nodes in a dispersed manner, improving the reliability of the data. Other data owners can obtain the corresponding shared shares of the intermediate results from the DHT as needed for subsequent global aggregation and analysis. The shared results flow between data owners and eventually converge to the MPC global calculation. By writing the shared shares of the intermediate results back to the DHT, the data owner completes the calculation of the local subtask and is ready for subsequent global calculations. The flow and convergence of shared results provide the necessary input for the MPC global calculation.

[0089] The MPC protocol is used to complete the global calculation and write the results back to the main chain. The local MMS shared results of each subtask distributed on the DHT are used as the input secret of the MPC. These shared results are generated by the data owner after executing the local subtask. Each participating node obtains the intermediate result sharing related to it from the DHT as the input secret. The participating node obtains the shared results it needs from the corresponding position of the DHT according to the predefined rules. The obtained shared results are used as the input secret of the node participating in the MPC global aggregation calculation. The input secret exists in an encrypted form to ensure the confidentiality of the data. MPC global aggregation, the global aggregation of subtask results is completed through the MPC protocol. The MPC protocol allows multiple parties to participate in the calculation and obtain the global result without leaking the original data. The blind calculation method is used for MPC aggregation. Each node encrypts the local shared results based on a random mask to generate an encrypted input secret. The encrypted input secret is aggregated during the execution of the MPC protocol. The aggregation calculation can be summation, averaging, voting and other operations, depending on the application scenario and requirements.

[0090] Interactive computing, the MPC aggregation process is implemented through multiple rounds of interactive computing. Participating nodes perform multiple rounds of communication and computing with other nodes in accordance with the provisions of the MPC protocol. Each round of computing is performed on encrypted data to ensure the confidentiality of the data. The nodes exchange encrypted intermediate results without leaking the original data. The purpose of multiple rounds of interactive computing is to gradually aggregate the encrypted data and finally obtain the ciphertext form of the global result while protecting the privacy of each node. Unmasking, after multiple rounds of interactive computing are completed, the participating nodes unmask. The unmasking process restores the encrypted data to the ciphertext form of the global result. The global result in ciphertext form is undecryptable for each node, ensuring the confidentiality of the result. Only the main chain can restore the final result plaintext using homomorphic decryption. Cross-chain writing back results, using cross-chain communication to write the global ciphertext results obtained by MPC calculation back to the main chain. Cross-chain communication ensures that the calculation results are securely transmitted from the shard chain to the main chain. After the main chain receives the global ciphertext result, it triggers the smart contract for result decryption and verification. The smart contract is responsible for processing and verifying the received ciphertext result.

[0091] Threshold signature, each data node performs a threshold signature on the hash value of the ciphertext result. Threshold signature requires a certain number of nodes to participate in the signature, and a single node cannot generate a valid signature independently. Threshold signature is used to verify the correctness and integrity of MPC execution. Only when a sufficient number of honest nodes participate in the signature can a valid threshold signature be generated.

[0092] Result verification and decryption, the main chain verifies the correctness and integrity of the MPC execution based on the threshold signature of the global result. The main chain smart contract checks whether the received threshold signature meets the predefined threshold requirements. After verification, the main chain uses homomorphic decryption to restore the final result plaintext. Homomorphic decryption ensures that only the main chain can decrypt and obtain the plaintext result, while the shard chain nodes cannot obtain the plaintext. The plaintext result on the main chain is the final output of the MPC global aggregation, representing the calculation result across multiple data owners. The main chain notifies each data node of the plaintext result through events. The event mechanism ensures timely notification and delivery of the results. After receiving the plaintext result, the data node can perform subsequent processing and application. For example, the data node can use the result for decision support, statistical analysis and other purposes.

Claims

1. A data management method for big data services, comprising: Collect data to be processed; The process of collecting data is recorded on the blockchain in the form of transactions, and zero-knowledge proof is used to verify the authenticity of the data while protecting privacy; Homomorphically encrypt the data after authenticity verification, then slice the encrypted data using data slicing, and store the sliced ​​data in multiple storage nodes in a distributed manner; Set up a heterogeneous multi-chain platform and distribute the processing tasks of stored data to different shard chains in the heterogeneous multi-chain platform for parallel execution; At the same time, a verifiable delay function (VDF) is deployed in each shard chain as one of the consensus algorithms of the shard chain. Based on the business needs of each shard chain, other consensus algorithms other than the VDF algorithm are selected while selecting the VDF consensus algorithm. Deploy the business logic of each shard chain on the corresponding shard chain in the form of smart contracts, and use the cross-chain communication protocol to interact with data across shard chains; Adopting multi-party secure computing (MPC), the data processing tasks assigned to each distribution chain are decomposed into multiple subtasks, and the subtasks are assigned to multiple data owners; Each data owner first secretly shares their homomorphically encrypted input data, and then only executes the assigned subtask locally; Finally, the calculation process of all subtasks is completed through the MPC protocol, and the global calculation results of data processing are obtained without leaking the original data of each data owner.

2. The data management method for big data service according to claim 1, characterized in that: The process of collecting data is recorded on the blockchain in the form of transactions, and zero-knowledge proof is used to verify the authenticity of the data while protecting privacy, including: The process of collecting data is recorded on the blockchain in the form of transactions, which include metadata of the time and location of data collection and the operator, as well as the hash value of the data, but not the original data; The data provider uses the zero-knowledge proof protocol to generate a data authenticity certificate for the collected original data; the data authenticity certificate is used to prove that the original data held by the data provider is consistent with the data corresponding to the hash value recorded on the blockchain, and that the data collection process meets the predefined requirements; The data user obtains the data authenticity certificate and the transaction information recorded on the blockchain, and by verifying the data authenticity certificate, without directly obtaining the original data, confirms the authenticity of the original data held by the data provider and the compliance of the data collection process based on the metadata and hash value contained in the transaction information; after the verification is passed, the data provider will perform homomorphic encryption on the verified original data to obtain the ciphertext data, and pass the ciphertext data to the data user.

3. The data management method for big data service according to claim 2, characterized in that: Data slicing is used to slice the encrypted data and store the sliced ​​data in multiple storage nodes, including: Perform homomorphic encryption on the original data after authenticity verification to obtain encrypted data, and use the encrypted data as a large secret S; Select a positive integer t as the threshold for secret sharing, and a positive integer n greater than t as the total number of shards. Randomly select a t-1 degree polynomial f(x) over the finite field GF(p) such that f(0) = S. The coefficients of the polynomial f(x) are the randomly selected t-1 elements. The polynomial f(x) is expressed as: f(x)=a0+a1x+a2x 2 +......+a t-1 x t-1 modp Among them, a0 = S, a1 to a t-1 are randomly selected coefficients; x is the independent variable, p is the modulus, and t is the order of the polynomial; Choose n different x i , calculate y i =f(x i ), get n fragments (x i ,y i ), each slice contains an x ​​value and a y value; Using the threshold signature mechanism, the signing parties consisting of multiple storage nodes jointly generate a public-private key pair (p k ,s k ), public key p k Public, private keys k Use secret sharing algorithm to split into n private key shards s ki , each storage node holds a private key shard; Distributed storage is used to store the obtained n data shards and n private key shards in multiple storage nodes. Each storage node stores one data shard (x i ,y i ) and a private key shard s ki ; Among them, the allocation of shards is random; The data user randomly selects at least t nodes from multiple storage nodes according to the secret sharing threshold t, and collects t data shards (x i ,y i ), and according to the (x i ,y i ) value, reconstruct the t-1 degree polynomial f(x) through the Lagrange interpolation formula, take f(0) as the secret S, and get the encrypted data; The data user selects t nodes from the t storage nodes of the collection shard as the signing participants, and uses the threshold signature algorithm. The t signing participants use their private key shards to sign. ki The restored encrypted data S is threshold signed to generate t partial signatures, and the t partial signatures are aggregated to generate a complete signature to verify the validity of the complete signature; after the signature verification is passed, the restored encrypted data S is decrypted using the decryption function of the homomorphic encryption algorithm to obtain the original data in plain text.

4. The data management method for big data service according to claim 3, characterized in that: Set up a heterogeneous multi-chain platform and distribute the processing tasks of stored data to different shard chains in the heterogeneous multi-chain platform for parallel execution, including: Construct a directed acyclic graph (DAG) of the subtasks to be processed. Each node in the DAG corresponds to a subtask, and the directed edges represent the data dependency between subtasks. According to the number of subtask nodes in the DAG, create at least as many shard chain instances as the number of subtasks to form a heterogeneous multi-chain platform; The mapping relationship between subtasks and shard chains is recorded using a distributed hash table DHT, with the subtask ID as the key and the shard chain ID as the value to build a mapping table from tasks to shard chains. The subtask ID is generated by the unique identifier of the corresponding subtask node in the DAG, and the shard chain ID is generated by the unique identifier of the corresponding shard chain instance. Traverse the DAG graph to obtain the predecessor task node set of the current subtask node. The predecessor task node is the input data source of the current subtask node. Combined with the computing power information of the shard chain where the predecessor task node is located recorded in the DHT mapping table, select the shard chain with the best computing power for the current subtask node, and write the shard chain deployment information of the current subtask node into the DHT mapping table; Convert the processing logic of each subtask into smart contract code, and deploy the smart contract code to the smart contract running environment of the corresponding shard chain based on the shard chain information determined by the DHT mapping table; when the smart contract is executed, query the storage location of the input data shard according to the DHT mapping table and initiate a cross-chain data read request; In the consensus mechanism of each shard chain, the Verifiable Delay Function (VDF) is fixed as one of the consensus algorithms. According to the computing power of the shard chain, at least one consensus algorithm is selected from Proof of Work (PoW), Proof of Stake (PoS), Delegated Proof of Stake (DPoS) and Byzantine Fault Tolerance (PBFT), and combined with VDF to form a hybrid consensus mechanism. For subtasks with data dependencies, after the upstream subtask completes the calculation, the calculation results are sent to the shard chain where the downstream subtask is located through the cross-chain communication protocol.

5. The data management method for big data service according to claim 4, characterized in that: Construct a directed acyclic graph (DAG) of the subtasks to be processed, including: Divide the ciphertext data to be processed into multiple data shards, each data shard contains a portion of the ciphertext data, and use a hash function to generate a hash value for the content of each data shard as a unique identifier of the corresponding data shard, recorded as a shard ID; Based on the data sharding, construct the MAP stage subtask, take each data shard as the input of a MAP subtask, and assign a unique task ID to each MAP subtask; the MAP subtask preprocesses the ciphertext data and generates intermediate result data; Process the intermediate result data of all generated MAP subtasks and construct Reduce stage subtasks; Construct a subtask dependency graph based on the input and output data relationship between the MAP subtask and the Reduce subtask; Build a DAG graph based on the constructed subtask dependency graph.

6. The data management method for big data service according to claim 5, characterized in that: In the consensus mechanism of each shard chain, VDF is combined to form a hybrid consensus mechanism, including: Deploy the hybrid consensus mechanism to the consensus node corresponding to the current shard chain as the consensus algorithm of the corresponding shard chain. During the blockchain synchronization process, the consensus node executes the hybrid consensus mechanism: The consensus node broadcasts the newly generated block to other consensus nodes and performs hybrid consensus calculations to obtain the block's timing proof and confirmation proof; The consensus nodes compare the calculated timing proof and confirmation proof. If they are consistent, the block is added to the local shard chain and the hybrid consensus execution result is synchronized. The global state of the shard chain is updated to complete the block synchronization. Among them, hybrid consensus calculation includes: The block header data of the block to be synchronized is used as the input of the VDF algorithm to obtain the timing proof of the block header; The complete data of the block to be synchronized is used as the input of at least one consensus algorithm selected from Proof of Work PoW, Proof of Stake PoS, Delegated Proof of Stake DPoS and Byzantine Fault Tolerance PBFT to obtain the confirmation proof of the block.

7. The data management method for big data service according to claim 6, characterized in that: For subtasks with data dependencies, after the upstream subtask completes the calculation, the calculation results are sent to the shard chain where the downstream subtask is located through the cross-chain communication protocol, including: Get the predecessor subtask set of the current subtask from DAG, and monitor the smart contract events of the shard chain where the predecessor subtask is located; After the upstream subtask is calculated, the shard chain where it is located calls the cross-chain communication contract and writes the calculation result to the specified storage location of the shard chain where the downstream subtask is located; After the cross-chain writing is completed, the cross-chain communication contract of the shard chain where the upstream subtask is located triggers a smart contract event to notify the shard chain where the downstream subtask is located; After the shard chain where the downstream subtask is located listens to the event, it accesses the specified storage location to read the calculation result of the upstream subtask, uses it as input to execute the local smart contract, and performs the calculation of the current subtask.

8. The data management method for big data service according to any one of claims 2 to 7, characterized in that: Using multi-party secure computing (MPC), we can obtain the global calculation results of data processing, including: The data processing tasks assigned to each shard chain are decomposed into multiple subtasks, and the subtasks are assigned to multiple data owners. The input data of the subtasks is obtained through the DHT mapping table, and the output results are written back to the DHT after secret sharing. Each data owner uses the secret sharing protocol MMS to perform secret sharing on the homomorphically encrypted input data. The sharing results are transmitted between data owners through DHT and used for local subtask calculations. Each data owner only executes the assigned subtask locally and writes the shared intermediate results back to the DHT for MPC global computing to read; The MPC protocol is used to complete the global calculation of all subtasks, and the generated local shares are used as input. The global ciphertext results of the MPC calculation are written back to the main chain through cross-chain communication. The main chain restores the plaintext results through homomorphic decryption and triggers a notification event.

9. The data management method for big data service according to claim 8, characterized in that: Each data owner only performs the assigned subtasks locally, including: Read the input data shard corresponding to the subtask assigned to this node from the DHT mapping table; Use the secret sharing protocol MMS to reconstruct the input password of the subtask, complete the local calculation, and apply the MMS protocol to the calculation result to generate the shared share of the intermediate result; The shared intermediate results are written back to the corresponding position of the DHT mapping table. The shared results flow between data owners through DHT and finally converge to MPC global calculation.

10. The data management method for big data service according to claim 9, characterized in that: The MPC multi-party secure computing protocol is used to obtain the global computing results of the data processing task without leaking the original data of each data owner, including: The generated local MMS sharing results of each subtask are used as the input secret of MPC, and the global aggregation of subtask results is completed through MPC; wherein, the input of MPC comes from the sharing of intermediate results distributed on DHT; The blind computing method is used for MPC aggregation. Each node encrypts the local shared results based on a random mask. After multiple rounds of interactive computing, the mask is removed to obtain the encrypted form of the global result. The global ciphertext results obtained by MPC calculation are written back to the main chain through cross-chain communication, triggering the result decryption and verification smart contract on the main chain; each data node performs a threshold signature on the hash value of the ciphertext result; The main chain verifies the correctness and integrity of MPC execution based on the threshold signature of the global result, uses homomorphic decryption to restore the final plaintext result, and notifies each data node of the plaintext result through events.

Images