Block chain on-chain and off-chain storage method and system based on file fragmentation

Abstract

The invention provides a block chain up-chain and down-chain storage method and system based on file fragments, and relates to the technical field of block chain storage, and the method comprises the steps: obtaining transaction encryption file data to be subjected to up-chain storage and a block chain node; fragmenting the transaction encrypted file data, classifying block chain nodes, and determining a main node for each class of block chain nodes; storing the fragmented transaction encryption file in the divided different types of block chain nodes, and confirming that contents sent by each type of block chain node are consistent through a consensus algorithm; calculating a hash value for each group of fragmented data for positioning, collecting storage fragments among different types of blockchain nodes by using an intelligent contract in a blockchain broadcast form, and determining correct fragmented file contents through a consensus mechanism so as to realize on-chain information verification; and completely splicing the storage fragments passing the on-chain information verification to obtain an initial transaction encrypted file data complete file, thereby realizing the on-chain distributed storage of the block chain.

Description

Technical Field

[0001] The present disclosure relates to the field of blockchain storage technology, and in particular to a blockchain on-chain and off-chain storage method and system based on file sharding. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Traditional blockchain technology is a distributed ledger technology that records transactions or information in a decentralized manner, and has the characteristics of being tamper-proof, highly transparent, and highly secure. Initially, blockchain was mainly used in the field of cryptocurrency, but people soon realized its potential application value in multiple industries such as finance, supply chain management, and copyright protection. Over time, many different blockchain platforms and technical improvements have emerged, such as Ethereum, which introduced smart contract functions and expanded the scope of blockchain applications.

[0004] At present, the conventional blockchain data storage mode is to package data such as transactions and smart contract status on the chain, and store all data completely on each node on the chain. However, in some scenarios, due to too frequent transactions, the following problems may occur:

[0005] 1) The amount of data on the chain is too large, and a single node needs to save too much data, causing node storage pressure; there is a performance bottleneck when processing a large number of transactions, and on-chain storage requires a large amount of storage space. The on-chain storage space is very precious, resulting in low blockchain execution efficiency.

[0006] 2) Certain consensus mechanisms (such as Proof of Work PoW) require a lot of energy consumption and have an impact on the environment.

[0007] 3) The current blockchain technology has problems such as the storage capacity of a single node is too large, and the data of each node, which is mainly based on the user's computer, is subject to security risks such as being decrypted by a single node. The encrypted files on the chain are at risk of being cracked. Due to the blockchain consensus mechanism, the behavior of each node cannot be controlled. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes a blockchain on-chain and off-chain storage method and system based on file sharding. Through the blockchain data security model of large file sharding, a block distributed storage mode is adopted for the stored data, which reduces the data storage pressure of each blockchain node, optimizes the consensus algorithm, and locates and traces each group of sharded data by recording hash values. During verification, the storage fragments between different personal nodes are collected and pieced together into a complete file, thereby realizing on-chain information verification.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions:

[0010] The blockchain on-chain and off-chain storage method based on file sharding includes:

[0011] Obtain the encrypted transaction file data and blockchain nodes to be stored on the chain;

[0012] The transaction encrypted file data is sharded, and the blockchain nodes are classified into categories, with a master node determined for each category of blockchain nodes;

[0013] The sharded encrypted transaction files are stored in different types of blockchain nodes, and the consensus algorithm is used to confirm that the content sent by each type of blockchain node is consistent;

[0014] For each group of shard data, the hash value is calculated for location, and the storage shards between different types of blockchain nodes are collected through blockchain broadcasting and smart contracts. The correct shard file content is determined through the consensus mechanism to achieve on-chain information verification;

[0015] The storage shards verified by the on-chain information are completely spliced ​​together to obtain the initial transaction encrypted file data complete file, realizing the on-chain distributed storage of the blockchain.

[0016] According to some embodiments, the present disclosure adopts the following technical solutions:

[0017] The blockchain on-chain and off-chain storage system based on file sharding includes:

[0018] The data acquisition module is used to obtain the encrypted transaction file data and blockchain nodes to be stored on the chain;

[0019] The sharding module is used to shard the transaction encrypted file data and classify the blockchain nodes into categories, with one master node for each category of blockchain nodes;

[0020] The consensus module is used to store the sharded encrypted transaction files in the different types of blockchain nodes, and confirm the consistency of the content sent by each type of blockchain node through the consensus algorithm;

[0021] The verification module is used to calculate the hash value for each group of shard data, collect the storage shards between different types of blockchain nodes through blockchain broadcasting, and use smart contracts to determine the correct shard file content through the consensus mechanism to achieve on-chain information verification;

[0022] The storage module is used to completely splice the storage shards that have passed the on-chain information verification to obtain the initial transaction encrypted file data complete file, thereby realizing the on-chain distributed storage of the blockchain.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] A computer program product includes a computer program, which, when executed by a processor, implements the blockchain on-chain and off-chain storage method based on file sharding.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the blockchain on-chain and off-chain storage method based on file sharding is implemented.

[0027] According to some embodiments, the present disclosure adopts the following technical solutions:

[0028] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the blockchain on-chain and off-chain storage method based on file sharding.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The disclosed on-chain and off-chain storage method of blockchain based on file sharding optimizes the traditional consensus algorithm through the large file sharding storage mode, and stores the encrypted large files required to be stored on the chain in the form of shards, and stores all the files that need to be stored in all nodes in the divided nodes of different categories. For each group of sharded data, the location and traceability are carried out by recording the hash value. During verification, the storage fragments between different personal nodes are collected and pieced together into a complete file, thereby realizing on-chain information verification; the storage pressure of each node is saved, and at the same time, because each node only stores part of the data, it also avoids some curious malicious nodes from snooping on some confidential storage data that have to be stored on the chain.

[0031] The disclosed on-chain and off-chain storage method of blockchain based on file sharding designs a unique consensus algorithm and an IPFS distributed sharding algorithm. The consensus algorithm divides the blocks that originally need to be agreed upon in the whole chain into several parts, and then only reaches consensus in the nodes corresponding to the part. Since each part of the block is smaller, the consensus efficiency will become higher. The sharding algorithm is to shard some large files that need to be stored on the chain. For example, a file of 1MB in size needs to be encrypted and packaged and uploaded to the blockchain in the form of blocks. Through the sharding algorithm, the encrypted file is divided into 4 small blocks of 256KB and sent to 4 different types of nodes respectively. The node types are divided into 4 categories in order of numbering, and distributed consensus can be performed, that is, each block that needs consensus is only reached in the owner node, which improves the consensus efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0033] Figure 1 This is a flow chart of the on-chain and off-chain trusted shard storage method according to an embodiment of the present disclosure;

[0034] Figure 2 Flow chart of the consensus algorithm of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] Example 1

[0039] In one embodiment of the present disclosure, a blockchain on-chain and off-chain storage method based on file sharding is provided, and the specific implementation steps include:

[0040] Step 1: Obtain the encrypted transaction file data and blockchain node to be stored on the chain;

[0041] Step 2: Shard the encrypted transaction file data and classify the blockchain nodes into categories, with one master node for each category of blockchain nodes.

[0042] Step 3: Store the sharded encrypted transaction files in different types of blockchain nodes, and use the consensus algorithm to confirm that the content sent by each type of blockchain node is consistent;

[0043] Step 4: Calculate the hash value for each group of shard data to locate it, broadcast it through the blockchain, and use smart contracts to collect storage shards between different types of blockchain nodes. Determine the correct shard file content through the consensus mechanism to achieve on-chain information verification;

[0044] Step 5: Completely splice the storage shards that have passed the on-chain information verification to obtain the initial transaction encrypted file data complete file, and realize the on-chain distributed storage of the blockchain.

[0045] As an embodiment, the specific implementation process of a blockchain on-chain and off-chain storage method based on file sharding disclosed in the present invention is as follows:

[0046] Step 1: Obtain the encrypted transaction file data and blockchain node to be stored on the chain;

[0047] Step 2: Shard the transaction encrypted file data and classify the blockchain nodes into categories, with one master node for each category of blockchain nodes;

[0048] Specifically, the transaction encrypted file data is sharded, including: file broadcast collection, packaging of the transaction encrypted file data to be stored on the chain, the sending node divides the file into N file pieces of a set size according to the size of the transaction encrypted file, and classifies the blockchain nodes. The N file pieces are transmitted to N different types of blockchain nodes on the chain.

[0049] Among them, after the transaction encrypted file data is packaged, all blockchain nodes are first divided, and the transaction encrypted file data is sharded, and the transaction encrypted file data is divided into N shards of 256KB in size (the part less than 256KB is filled with zeros). At the same time, the nodes on the chain are divided into N different categories. These N file pieces will be transmitted to N different types of users on the chain. Each shard file data needs to add the shard hash, the assigned node number, the remaining shard hashes and the total number of file shards. Through the sharding form, the data that originally needs to be stored in all nodes is reduced to only part of the data, saving the storage pressure of each node. At the same time, since each node only stores part of the data, it also reduces the confidential files on the chain from being cracked by some malicious nodes.

[0050] Furthermore, the nodes on the chain are divided into N categories, each category of nodes determines a master node, and the corresponding N file pieces are sent to these N categories respectively, and the consensus algorithm is used to confirm that the content sent by each category of users is consistent.

[0051] Step 3: File acceptance: store the sharded encrypted transaction files in different types of blockchain nodes, and use the consensus algorithm to confirm that the content sent by each type of blockchain node is consistent;

[0052] Among them, the consensus algorithm is used to confirm that the content sent by each type of user is consistent, including:

[0053] The blockchain nodes are classified into N categories, each category of blockchain nodes determines a master node, and N file pieces are sent to the N categories of blockchain nodes respectively, and the consensus algorithm is used to confirm that the content sent by each category of users is consistent. The consensus algorithm selects a master node from each category of blockchain nodes, and then reaches a consensus through three-stage voting.

[0054] Among them, the three-stage voting includes the pre-preparation stage, the preparation stage and the submission stage.

[0055] As an example, the specific process of reaching consensus through three-stage voting is as follows:

[0056] Pre-preparation stage: select a master node from each type of blockchain node according to the file slicing situation; the node that needs to store large files on the chain broadcasts to the entire network, the master node collects all transaction information, creates a new block and broadcasts it to the entire network;

[0057] Preparation phase: After receiving the file slice sent by its master node, each node enters the preparation phase from the pre-preparation phase. The master node verifies the file slice. After the verification is passed, it broadcasts a preparation message to other nodes under it and sends the file slice.

[0058] Commit phase: After broadcasting the prepare message to the entire network, the node enters the commit phase. If the node receives prepare messages from more than 2 / 3 of the nodes, it broadcasts a commit message to the entire network. If a node receives commit messages from more than 2 / 3 of the nodes, it can submit the new block to the local blockchain and reach a consensus on the latest height block.

[0059] Step 4: File voting: For each group of shard data, calculate the hash value to locate it, broadcast it through the blockchain, and use smart contracts to collect storage shards between different types of blockchain nodes. Determine the correct shard file content through the consensus mechanism to achieve on-chain information verification;

[0060] Specifically, when the required content needs to be verified or viewed, the master node that needs to be verified collects data in the form of each shard of all online nodes through blockchain broadcasting and using smart contracts, and determines whether the uploaded file is correct by accepting the hash value of the uploaded file. When there is a difference in the data uploaded by a certain file shard, the master node votes to verify the hash of all files uploaded by the shard, and determines the correct shard file content through a consensus mechanism.

[0061] As an example, if a shard has 5 owners, when verification is required, the master node receives 5 shard files after broadcasting. Assuming that two different hash values ​​are obtained by hashing these 5 files, the two different hash values ​​are voted and counted to see which hash value has a larger number. The larger one is the consensus file slice. Then the remaining incorrect files are modified, and the correct file slice is resent to the 5 owners to complete the consensus.

[0062] Step 5: File reorganization: Completely splice the storage shards verified by the on-chain information to obtain the initial transaction encrypted file data complete file, realizing the on-chain distributed storage of the blockchain.

[0063] All file pieces will be combined to obtain a complete file.

[0064] Example 2

[0065] In one embodiment of the present disclosure, a blockchain on-chain and off-chain storage system based on file sharding is provided, including:

[0066] The data acquisition module is used to obtain the encrypted transaction file data and blockchain nodes to be stored on the chain;

[0067] The sharding module is used to shard the transaction encrypted file data and classify the blockchain nodes into categories, with one master node for each category of blockchain nodes;

[0068] The consensus module is used to store the sharded encrypted transaction files in the different types of blockchain nodes, and confirm the consistency of the content sent by each type of blockchain node through the consensus algorithm;

[0069] The verification module is used to calculate the hash value for each group of shard data, collect the storage shards between different types of blockchain nodes through blockchain broadcasting, and use smart contracts to determine the correct shard file content through the consensus mechanism to achieve on-chain information verification;

[0070] The storage module is used to completely splice the storage shards that have passed the on-chain information verification to obtain the initial transaction encrypted file data complete file, thereby realizing the on-chain distributed storage of the blockchain.

[0071] Example 3

[0072] In one embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the blockchain on-chain and off-chain storage method based on file sharding.

[0073] Example 4

[0074] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the blockchain on-chain and off-chain storage method based on file sharding is implemented.

[0075] Example 5

[0076] In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the blockchain on-chain and off-chain storage method based on file sharding.

[0077] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0079] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A blockchain on-chain and off-chain storage method based on file sharding, characterized in that: include: Obtain the encrypted transaction file data and blockchain nodes to be stored on the chain; The transaction encrypted file data is sharded, and the blockchain nodes are classified into categories, with a master node determined for each category of blockchain nodes; The sharded encrypted transaction files are stored in different types of blockchain nodes, and the consensus algorithm is used to confirm that the content sent by each type of blockchain node is consistent; For each group of shard data, the hash value is calculated for location, and the storage shards between different types of blockchain nodes are collected through blockchain broadcasting and smart contracts. The correct shard file content is determined through the consensus mechanism to achieve on-chain information verification; The storage shards verified by the on-chain information are completely spliced ​​together to obtain the initial transaction encrypted file data complete file, realizing the on-chain distributed storage of the blockchain.

2. The blockchain on-chain and off-chain storage method based on file sharding as claimed in claim 1, characterized in that: The transaction encrypted file data is fragmented, including: the transaction encrypted file data to be stored on the chain is packaged, the sending node divides the file into N file pieces of a set size according to the size of the transaction encrypted file, and the blockchain nodes are classified, and the N file pieces are transmitted to N different types of blockchain nodes on the chain.

3. The blockchain on-chain and off-chain storage method based on file sharding as claimed in claim 1, characterized in that: Add the shard hash, allocation node number, remaining shard file hashes, and total file shard data information to each set of shard file data.

4. The blockchain on-chain and off-chain storage method based on file sharding as claimed in claim 1, characterized in that: Blockchain nodes are classified into categories, and each category of blockchain nodes has a master node, including: The blockchain nodes are classified into N categories, each category of blockchain nodes determines a master node, and N file pieces are sent to the N categories of blockchain nodes respectively, and the consensus algorithm is used to confirm that the content sent by each category of users is consistent. The consensus algorithm selects a master node from each category of blockchain nodes, and then reaches a consensus through three-stage voting.

5. The blockchain on-chain and off-chain storage method based on file sharding as claimed in claim 4 is characterized in that: The three-stage voting includes a pre-preparation stage, a preparation stage, and a submission stage. The pre-preparation stage includes selecting a master node from each type of blockchain node according to the file sharding situation, the file chain node broadcasts to the entire network, the master node collects all transaction information, creates a new block and broadcasts it to the entire network, and the preparation stage includes each blockchain node entering the preparation stage from the pre-preparation stage after receiving the file piece sent by its master node. The master node verifies the file piece, broadcasts a preparation message to other nodes after the verification is passed, and sends the file piece; The commit phase includes a node entering the commit phase after broadcasting a prepare message to the entire network. If a node receives prepare messages from more than 2 / 3 of the nodes, it broadcasts a commit message to the entire network. If a node receives submission messages from more than 2 / 3 of the nodes, it can submit the new block to the local blockchain and reach a consensus on the latest height block.

6. The blockchain on-chain and off-chain storage method based on file sharding as claimed in claim 1, characterized in that: To verify the file content, the master node that needs to be verified collects data in each shard form of all online nodes through blockchain broadcasting and using smart contracts. It determines whether the uploaded file is correct by accepting the hash value of the uploaded file. When there is a difference in the data uploaded by a certain file shard, the master node votes to verify the hash of all files uploaded by the shard, and determines the correct shard file content through a consensus mechanism.

7. The blockchain on-chain and off-chain storage system based on file sharding is characterized by: include: The data acquisition module is used to obtain the encrypted transaction file data and blockchain nodes to be stored on the chain; The sharding module is used to shard the transaction encrypted file data and classify the blockchain nodes into categories, with one master node for each category of blockchain nodes; The consensus module is used to store the sharded encrypted transaction files in the different types of blockchain nodes, and confirm the consistency of the content sent by each type of blockchain node through the consensus algorithm; The verification module is used to calculate the hash value for each group of shard data, collect the storage shards between different types of blockchain nodes through blockchain broadcasting, and use smart contracts to determine the correct shard file content through the consensus mechanism to achieve on-chain information verification; The storage module is used to completely splice the storage shards that have passed the on-chain information verification to obtain the initial transaction encrypted file data complete file, thereby realizing the on-chain distributed storage of the blockchain.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the on-chain and off-chain storage method of blockchain based on file sharding according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the blockchain on-chain and off-chain storage method based on file sharding as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the blockchain on-chain and off-chain storage method based on file sharding as described in any one of claims 1-6.

Images