Data processing method and device based on block chain, equipment and medium

Abstract

The embodiment of the invention discloses a data processing method and device based on a block chain, equipment and a medium. The method comprises the steps of obtaining an archived file containing a linked block and an index file containing a key-value pair corresponding to the linked block based on a data snapshot request, searching a latest key-value file corresponding to the same key element from the index file, then generating a snapshot file based on the key-value file and the archived file, and storing the snapshot file in the database. Generating a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file, and carrying out associated storage on the snapshot file and the snapshot verification file; the method comprises the following steps: acquiring a snapshot file and a snapshot verification file which are stored in an associated manner based on a data import request, and if the snapshot file passes verification based on the snapshot verification file, acquiring a key value file and an archiving file from the snapshot file, and importing the key value file and the archiving file into a local storage area. According to the technical scheme, the reliability of data snapshot and data import in the block chain is improved.

Description

Technical Field

[0001] The present application relates to the field of blockchain technology, and specifically, to a blockchain-based data processing method, a blockchain-based data processing device, an electronic device, and a computer-readable medium. Background Art

[0002] It is understandable that during the operation of the blockchain, the block data may be snapshotted according to actual needs, so as to complete the data import through the obtained snapshot data. In the related art, the snapshot party (for example, a blockchain node in the same blockchain) packages all the block data in the local storage area after stopping the block from being uploaded to the chain, thereby completing the snapshot of the block data. However, the snapshot data obtained in this way is large in size, occupies storage space, and has low snapshot efficiency. At the same time, there may be phenomena such as snapshot data being tampered with, and the security is low.

[0003] Therefore, how to improve the reliability of data snapshots in blockchain to improve the reliability of data import is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a blockchain-based data processing method, device, equipment, and medium, which improve the reliability of data snapshots and data import in the blockchain.

[0005] In a first aspect, an embodiment of the present application provides a data processing method based on blockchain, the method comprising: if a data snapshot request is received, obtaining an archive file containing chained blocks and an index file containing key-value pairs corresponding to the chained blocks, the key-value pairs comprising a key element and a value element; searching the index file for the latest value element corresponding to the same key element, and generating a key-value file corresponding to the index file based on the key-value pair matching the latest value element; generating a snapshot file based on the key-value file and the archive file, and generating a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file; and storing the snapshot file and the snapshot verification file in association with each other.

[0006] In a second aspect, an embodiment of the present application provides a data processing method based on blockchain, the method comprising: if a data import request is received, obtaining an associated stored snapshot file and a snapshot verification file; wherein the snapshot file comprises a key-value file and an archive file, the key-value file being generated based on a key-value pair that matches the latest value element corresponding to the same key element; if the verification of the snapshot file based on the snapshot verification file passes, obtaining the key-value file and the archive file from the snapshot file; and importing the key-value file and the archive file into a local storage area.

[0007] In a third aspect, an embodiment of the present application provides a data processing device based on blockchain, the device comprising: an acquisition module, configured to obtain an archive file containing chained blocks and an index file containing key-value pairs corresponding to the chained blocks if a data snapshot request is received, the key-value pairs comprising a key element and a value element; a first generation module, configured to search the index file for the latest value element corresponding to the same key element, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element; a second generation module, configured to generate a snapshot file based on the key-value file and the archive file, and generate a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file; a storage module, configured to store the snapshot file and the snapshot verification file in association.

[0008] In a fourth aspect, an embodiment of the present application provides a blockchain-based data processing device, the device comprising: a first acquisition module, configured to obtain an associated stored snapshot file and a snapshot verification file if a data import request is received; wherein the snapshot file comprises a key-value file and an archive file, the key-value file being generated based on a key-value pair that matches the latest value element corresponding to the same key element; a second acquisition module, configured to obtain the key-value file and the archive file from the snapshot file if the verification of the snapshot file based on the snapshot verification file passes; an import module, configured to import the key-value file and the archive file into a local storage area.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the blockchain-based data processing method as described above.

[0010] In a sixth aspect, an embodiment of the present application provides a computer-readable medium on which a computer program is stored, and when the computer program is executed by a processor, the blockchain-based data processing method as described above is implemented.

[0011] In a seventh aspect, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the blockchain-based data processing method as described above.

[0012] In the technical solution provided in the embodiments of the present application:

[0013] On the one hand, the latest value element corresponding to the same key element is searched from the index file, and a key-value file corresponding to the index file is generated based on the key-value pair matching the latest value element. The generated key-value file has a small amount of data, and accordingly, the generated snapshot file has a small amount of data, which saves storage space and transmission time, thereby improving the efficiency of data snapshots and data import in the blockchain.

[0014] On the one hand, when a snapshot file is generated based on the key-value file and the archive file, a snapshot verification file is generated at the same time. The snapshot file can be verified using the snapshot verification file, and data can be imported after the verification passes, thereby improving the security of data snapshots and data import in the blockchain.

[0015] In summary, the technical solution provided by the embodiments of the present application has high reliability of data snapshot and data import.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A It is a structural diagram of the data sharing system.

[0018] Figure 1B It is a schematic diagram of the connection relationship between blocks in the blockchain.

[0019] Figure 1C This is a schematic diagram of the generation of a new block.

[0020] Figure 2 It is a schematic diagram of an exemplary implementation environment to which the technical solution of the embodiments of the present application can be applied.

[0021] Figure 3 It is a flowchart of a blockchain-based data processing method shown in an exemplary embodiment of the present application.

[0022] Figure 4 It is a flowchart of a blockchain-based data processing method shown in another exemplary embodiment of the present application.

[0023] Figure 5 It is a schematic diagram of a log structure merge tree shown in an exemplary embodiment of the present application.

[0024] Figure 6 It is a flowchart of a blockchain-based data processing method shown in another exemplary embodiment of the present application.

[0025] Figure 7 It is a flowchart of a blockchain-based data processing method shown in another exemplary embodiment of the present application.

[0026] Figure 8 It is a flowchart of a blockchain-based data processing method shown in another exemplary embodiment of the present application.

[0027] Fig. 9 It is a flowchart of a blockchain-based data processing method shown in another exemplary embodiment of the present application.

[0028] Fig.10 It is a flowchart of a blockchain-based data processing method shown in an exemplary embodiment of the present application.

[0029] Fig.11 It is a flowchart of a blockchain-based data processing method shown in another exemplary embodiment of the present application.

[0030] Fig.12 It is a flowchart of a blockchain-based data processing method shown in another exemplary embodiment of the present application.

[0031] Fig.13 It is a schematic diagram of the overall architecture shown by an exemplary embodiment of the present application.

[0032] Fig.14 It is a schematic diagram of a block in a xxx.fdb file shown in an exemplary embodiment of the present application.

[0033] Fig.15 It is a schematic diagram of the storage of archive files and index files shown in an exemplary embodiment of the present application.

[0034] Fig.16 It is a schematic diagram of snapshot file storage shown in an exemplary embodiment of the present application.

[0035] Fig.17 It is a schematic diagram of multiple slice areas shown in an exemplary embodiment of the present application.

[0036] Fig.18 It is a schematic diagram of the key range of the xxx.csv file to be newly generated, shown in an exemplary embodiment of the present application.

[0037] Fig.19 It is a block diagram of a blockchain-based data processing device shown as an exemplary embodiment of the present application.

[0038] Fig. 20 It is a block diagram of a blockchain-based data processing device shown as an exemplary embodiment of the present application.

[0039] Fig.21 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0040] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments that are the same as the present application. Instead, they are only examples of devices and methods that are the same as some aspects of the present application as detailed in the attached claims.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0043] It should be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0044] Before introducing the technical solution of the embodiment of the present application, the blockchain technology used in the embodiment of the present application is first introduced.

[0045] See also Figure 1A The data sharing system shown, the data sharing system 100 refers to a system for sharing data between nodes, and the data sharing system may include multiple nodes 101, and the multiple nodes 101 may refer to each client in the data sharing system. Each node can receive input information during normal operation, and maintain the shared data in the data sharing system based on the received input information. In order to ensure the information intercommunication within the data sharing system, there may be an information connection between each node in the data sharing system, and information can be transmitted between nodes through the above information connection. For example, when any node in the data sharing system receives input information, other nodes in the data sharing system obtain the input information according to the consensus algorithm, and store the input information as data in the shared data, so that the data stored on all nodes in the data sharing system are consistent.

[0046] Each node in the data sharing system has a corresponding node identifier, and each node in the data sharing system can store the node identifiers of other nodes in the data sharing system, so that the generated blocks can be broadcast to other nodes in the data sharing system according to the node identifiers of other nodes. Each node can maintain a node identifier list as shown in the following table, and store the node name and node identifier in the node identifier list accordingly. The node identifier can be an IP (Internet Protocol, a protocol for interconnecting networks) address and any other information that can be used to identify the node.

[0047] Each node in the data sharing system stores the same blockchain. The blockchain consists of multiple blocks, see Figure 1B The blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores the input information characteristic value, version number, timestamp and difficulty value, and the block body stores the input information; the next block of the genesis block takes the genesis block as the parent block, and the next block also includes a block header and a block body. The block header stores the input information characteristic value of the current block, the block header characteristic value, version number, timestamp and difficulty value of the parent block, and so on, so that the block data stored in each block in the blockchain is associated with the block data stored in the parent block, ensuring the security of the input information in the block.

[0048] When generating each block in the blockchain, see Figure 1C When the node where the blockchain is located receives the input information, it verifies the input information. After the verification is completed, the input information is stored in the memory pool and the hash tree used to record the input information is updated. After that, the update timestamp is updated to the time when the input information is received, and different random numbers are tried to calculate the eigenvalue multiple times so that the calculated eigenvalue can satisfy the following formula:

[0049] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x)) <TARGET

[0050] Among them, SHA256 is the eigenvalue algorithm used to calculate the eigenvalue; version (version number) is the version information of the relevant block protocol in the blockchain; prev_hash is the block header eigenvalue of the parent block of the current block; merkle_root is the eigenvalue of the input information; ntime is the update time of the update timestamp; nbits is the current difficulty, which is a fixed value within a period of time and is determined again after exceeding the fixed time period; x is a random number; TARGET is the eigenvalue threshold, which can be determined based on nbits.

[0051] In this way, when the random number that satisfies the above formula is calculated, the information can be stored accordingly, the block header and block body can be generated, and the current block can be obtained. Subsequently, the node where the blockchain is located sends the newly generated block to other nodes in the data sharing system according to the node identification of other nodes in the data sharing system. Other nodes verify the newly generated block and add the newly generated block to the blockchain stored in them after the verification is completed.

[0052] It is understandable that during the operation of the blockchain, the block data may be snapshotted according to actual needs, so as to complete the data import through the obtained snapshot data. In the related art, the snapshot party (for example, a blockchain node in the same blockchain) packages all the block data in the local storage area after stopping the block from being uploaded to the chain, thereby completing the snapshot of the block data. However, the snapshot data obtained in this way is large in size, occupies storage space, and there may be phenomena such as snapshot data being tampered with, and the security is relatively low.

[0053] Therefore, in order to improve the reliability of data snapshots and data import in the blockchain, this application provides a data processing solution based on the blockchain. Figure 2 , Figure 2 2 is a schematic diagram of an implementation environment involved in the present application. The implementation environment mainly includes a snapshot party 201 and an importer 202. It can be understood that the snapshot party 201 and the importer 202 are connected via a network communication, and the network can include various connection types, such as wireless communication links, wired and optical fiber cables, etc., wherein:

[0054] The snapshot party refers to the party used to perform data snapshot / backup, which can be any node in the data sharing system.

[0055] The importer refers to the party that uses snapshot data for data import / synchronization. After completing the data import, it can join the data sharing system to become a member of the data sharing system, or it can itself be any node in the data sharing system. Completing the data import is used to achieve data repair, improvement, etc.

[0056] Among them, the snapshot party and / or importer can be a server that provides various services; servers that provide various services include but are not limited to independent physical servers, server clusters or distributed systems composed of multiple physical servers, cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The snapshot party and / or importer can also be a terminal device; terminal devices include but are not limited to smart phones, tablets, laptops, computers, intelligent voice interaction devices, smart home appliances, smart wearable devices, aircraft, etc.

[0057] In one embodiment of the present application, the blockchain-based data processing method can be interactively executed by the snapshot party and the importing party.

[0058] Exemplarily, if the snapshot party receives a data snapshot request, it obtains an archive file containing the chained blocks and an index file containing key-value pairs corresponding to the chained blocks, the key-value pairs including a key element and a value element; then, it searches the index file for the latest value element corresponding to the same key element, and generates a key-value file corresponding to the index file based on the key-value pair matching the latest value element; then, it generates a snapshot file based on the key-value file and the archive file, and generates a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file; then, the snapshot file and the snapshot verification file are associated and stored.

[0059] Exemplarily, if the importing party receives a data import request, it obtains the associated stored snapshot file and snapshot verification file from the snapshot party; then, if the snapshot file is verified based on the snapshot verification file, it obtains the key-value file and archive file from the snapshot file; then, it imports the key-value file and archive file into the local storage area.

[0060] Figure 2 The technical solutions of the illustrated embodiments can be applied to various scenarios, including but not limited to smart transportation, assisted driving, cloud technology, artificial intelligence, etc.; in actual applications, corresponding adjustments can be made according to specific application scenarios.

[0061] It should be noted that in the specific implementation of this application, user-related data is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0062] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0063] See also Figure 3 , Figure 32 is a flowchart of a data processing method based on blockchain shown in an embodiment of the present application. The data processing method based on blockchain can be executed by the snapshot party 201. Figure 3 As shown, the data processing method based on blockchain includes at least S301 to S304, which are described in detail as follows:

[0064] S301, if a data snapshot request is received, an archive file containing the chained blocks and an index file containing key-value pairs corresponding to the chained blocks are obtained, where the key-value pairs include a key element and a value element.

[0065] In an embodiment of the present application, when the requesting party has a data snapshot requirement, a data snapshot request can be issued, and then the data snapshot request can be sent to the snapshot party; accordingly, the snapshot party receives the data snapshot request sent by the requesting party, and can obtain an archive file containing the chained blocks, and an index file containing key-value pairs corresponding to the chained blocks, where the key-value pairs include a key element and a value element.

[0066] In the embodiment of the present application, the data snapshot request refers to a request for instructing to take a data snapshot.

[0067] In the embodiment of the present application, the archive file refers to a file used to archive the chained blocks, which includes a specified number (one or more) of chained blocks. Among them, the archive file can be represented by the xxx.fdb file, for example, the xx1.fdb file stores 5 blocks with block heights of 0, 1, 2, 3, and 4, and the xx2.fdb file stores 5 blocks with block heights of 5, 6, 7, 8, and 9. Exemplarily, the archive file can be stored in a block file storage database (BFDB for short).

[0068] In the embodiment of the present application, the index file refers to a file for storing key-value pairs converted from index data and status data related to the chained blocks. Among them, the index file can be represented by the xxx.ldb file, and the key-value pair includes a key element and a value element. The K in the key-value pair is the key element, and the V in the key-value pair is the value element. Exemplarily, the index file can be stored in a distributed KV database (abbreviated as KVDB), where KVDB includes but is not limited to leveldb and rocksdb (an optimized version of leveldb).

[0069] S302, searching the index file for the latest value element corresponding to the same key element, and generating a key-value file corresponding to the index file based on the key-value pair matching the latest value element.

[0070] In the embodiment of the present application, the snapshot party obtains the index file, and then can search the index file for the latest value element corresponding to the same key element, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element.

[0071] It is understandable that the index file stores multiple value elements corresponding to the same key element, which occupies a relatively large amount of storage space. However, usually the value elements of the old version (i.e., the value elements with a longer writing duration) are not used. Therefore, in the embodiments of the present application, instead of directly copying the index file, the latest value element corresponding to the same key element is searched from the index file, and a key-value file corresponding to the index file is generated based on the key-value pair matching the latest value element.

[0072] In the embodiments of the present application, the latest value element refers to the latest value element for the key element, which is newer than other value elements and has a shorter writing duration (the writing duration refers to the difference between the writing time and the current time); the key-value pair matching the latest value element refers to the key-value pair where the latest value element is located.

[0073] For example, referring to Table 1, it is an example of multiple value elements corresponding to the key element.

[0074]

[0075]

[0076] Table 1

[0077] In Table 1, t1_1 < t1_2 < t1_3, where < means later than. As shown in Table 1, for the key element xx_1, there are 3 value elements xx1_1, xx1_2, and xx1_3 respectively. Since the writing time of xx1_1 is the shortest, xx1_1 is the latest value element among the 3 value elements, that is, the latest value element corresponding to xx_1 is xx1_1; correspondingly, the key-value pair matching the latest value element xx1_1 is (xx_1, xx1_1).

[0078] In the embodiments of the present application, the key-value file refers to searching for the latest value element corresponding to each key element from the index file, and generating a key-value file corresponding to the index file based on the key-value pair matching each latest value element. Among them, the key-value file can be represented by a xxx.csv file.

[0079] For example, referring to Table 2, it is an example of the key-value file.

[0080] Key element (K) Value element (V) xx_1 xx1_1 xx_2 xx2_1 xx_3 xx3_2 …… ……

[0081] Table 2

[0082] As shown in Table 2, the key-value file includes multiple key-value pairs, and each value element in the multiple key-value pairs is the latest.

[0083] Therefore, a key-value file is generated by screening the key-value pairs contained in the index file. The generated key-value file does not contain old value elements, which can reduce the data volume of the old value elements, making the subsequently generated snapshot data (i.e., snapshot file) small in data volume and occupying less storage space.

[0084] In the embodiment of the present application, the key-value file corresponding to the index file means that the key-value file includes all the key elements in the index file and the latest value elements corresponding to the key elements. When the index file includes multiple key-value files, the number of key-value files can be the same as the number of index files or different.

[0085] S303, generating a snapshot file based on the key-value file and the archive file, and generating a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file.

[0086] In the embodiment of the present application, the snapshot party generates a key-value file, and then generates a snapshot file based on the key-value file and the archive file, and generates a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file.

[0087] In the embodiment of the present application, the snapshot file is the snapshot data, which includes a key-value file and an archive file; for example, F represents the snapshot file, then F = {xxx.csv, xxx.fdb}.

[0088] In the embodiment of the present application, the snapshot verification file refers to a file generated using the characteristic information / attribute information of the snapshot file for verifying the snapshot file. The characteristic information / attribute information of the snapshot file includes but is not limited to the hash data of the snapshot file.

[0089] In one embodiment of the present application, the process of generating a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file in S303 may include:

[0090] Compress the snapshot file, and perform a hash operation on the compressed snapshot file to obtain hash data corresponding to the compressed snapshot file;

[0091] A snapshot verification file for verifying the snapshot file is generated based on the hash data corresponding to the snapshot file.

[0092] That is, in an optional embodiment, the snapshot party compresses the snapshot file, performs a hash operation on the compressed snapshot file to obtain hash data corresponding to the compressed snapshot file, and then generates a snapshot verification file based on the hash data corresponding to the snapshot file.

[0093] Among them, in an optional embodiment, any compression algorithm can be used to compress the snapshot file to obtain a compressed snapshot file; at the same time, any hash algorithm can be used to perform a hash operation on the compressed snapshot file to obtain hash data corresponding to the compressed snapshot file.

[0094] S304: associate and store the snapshot file and the snapshot verification file.

[0095] In the embodiment of the present application, the snapshot party generates a snapshot file and a snapshot verification file, and then the snapshot file and the snapshot verification file can be associated and stored.

[0096] Exemplarily, the snapshot file and the snapshot verification file can be associated and stored in a data warehouse, where the data warehouse can refer to a database dedicated to hosting snapshot data (for example, one data warehouse corresponds to each blockchain, or multiple blockchains share one data warehouse), or it can refer to any database (for example, it can store other data besides snapshot data), etc. In actual applications, the data warehouse can be flexibly adjusted according to the specific application scenario.

[0097] In one embodiment of the present application, if the snapshot file is compressed, the compressed snapshot file and the snapshot verification file are stored in association with each other.

[0098] In the embodiment of the present application, the snapshot file generated by the snapshot party has a small amount of data, which saves storage space and transmission time, improves the efficiency of data snapshots in the blockchain, and also generates a snapshot verification file, which can be used by the importing party to verify the snapshot file, thereby improving the security of data snapshots in the blockchain.

[0099] In one embodiment of the present application, another blockchain-based data processing method is provided, which can be executed by the snapshot party 201. Figure 4 As shown, the blockchain-based data processing method may include S401, S301, S303 to S304.

[0100] In the embodiment of the present application, the key-value pairs contained in the index file are stored in the storage area in the form of a log-structured merge tree (LSM Tree).

[0101] See also Figure 5The log structure merge tree is a forest of multiple "subtrees" spanning memory and disk. It can be divided into Level 0, Level 1, Level 2...Level n subtrees (also called multiple merge layers). Only the Level 0 subtree is in memory, and the remaining Level 1 to Level n subtrees are in disk. The Level 0 subtree in memory generally uses an ordered data structure such as a sorted tree (red-black tree / AVL tree), a skip list, or a TreeMap to facilitate subsequent sequential writing to disk. The Level 1 to Level n subtrees on disk are essentially files that are written to disk in sequence after the data is sorted. Usually, each layer of subtrees has a storage threshold size. When the storage threshold is reached, they will be merged, and the merged results will be written to the next layer. Only data in memory is allowed to be updated in place. Changes to data on disk are only allowed to be appended, and no in-place updates are made.

[0102] Understandably, Figure 5 Only 4 subtrees / 4 merged layers are shown. In actual applications, it depends on the specific situation of data storage.

[0103] S401 is described in detail as follows:

[0104] S401, searching for the latest value element corresponding to the same key element from the storage area in descending order of the number of merge layers; wherein the number of merge layers is inversely proportional to the writing time of the key-value pair.

[0105] In the embodiment of the present application, the snapshot party obtains the index file, and then can search the storage area for the latest value element corresponding to the same key element in the order of the number of merge layers from high to low.

[0106] In the embodiment of the present application, the number of merged layers is inversely proportional to the write time of the key-value pairs; as mentioned above, the higher the number of merged layers, the newest key-value pairs, among which Level 0 is the highest merged layer, and the key-value pairs it contains have the shortest write time, and accordingly, the key-value pairs are the newest, and Level n is the lowest merged layer, and the key-value pairs it contains have the longest write time, and accordingly, the key-value pairs are the oldest.

[0107] It should be noted that Figure 4 For detailed description of S301, S303 to S304, please refer to Figure 3 S301, S303 to S304 shown are not described in detail here.

[0108] In the embodiment of the present application, the snapshot party searches the storage area for the latest value element corresponding to the same key element in the order of the number of merge layers from high to low, and the search efficiency is high. Accordingly, the efficiency of generating key-value files is improved, thereby improving the efficiency of data snapshots.

[0109] In one embodiment of the present application, another blockchain-based data processing method is provided, which can be executed by the snapshot party 201. Figure 6 As shown, the blockchain-based data processing method may include S601 to S603, S301 to S302, and S304.

[0110] S601 to S603 are described in detail as follows:

[0111] S601, compress the key-value file, and perform a hash operation on the compressed key-value file to obtain hash data corresponding to the compressed key-value file.

[0112] In the embodiment of the present application, the snapshot party generates a key-value file, which can then be compressed, and a hash operation is performed on the compressed key-value file to obtain hash data corresponding to the compressed key-value file.

[0113] For example, continuing with the previous example, use xxx.csv file to represent the key-value file, wherein the xxx.csv file can be compressed by 7z compression algorithm to obtain xxx.csv.7z file, and then the hash algorithm is used to perform hash operation on the xxx.csv.7z file, i.e. Hash(xxx.csv.7z file), to obtain the hash data hash_xxx.csv.7z of the xxx.csv.7z file.

[0114] S602: Generate a key-value verification file for verifying the key-value file based on the hash data corresponding to the key-value file.

[0115] In the embodiment of the present application, the snapshot party obtains the hash data corresponding to the key-value file, and then can generate a key-value verification file based on the hash data corresponding to the key-value file.

[0116] In the embodiment of the present application, the key-value verification file refers to a file generated by using the characteristic information / attribute information of the key-value file for verifying the key-value file, and the characteristic information / attribute information of the key-value file includes but is not limited to the hash data of the key-value file. Among them, the key-value verification file can be represented by a xxx.csv.txt file.

[0117] For example, continuing with the previous example, a key-value verification file xxx.csv.txt is generated based on hash_xxx.csv.7z.

[0118] S603, generating a snapshot file based on the archive file, the compressed key-value file, and the key-value verification file.

[0119] In the embodiment of the present application, the snapshot party obtains a compressed key-value file and a key-value verification file, and then can generate a snapshot file based on the archive file, the compressed key-value file, and the key-value verification file.

[0120] In the embodiment of the present application, the snapshot file includes an archive file, a compressed key-value file, and a key-value verification file; for example, continuing the above example, F represents the snapshot file, then F = {xxx.fdb, xxx.csv.7z, xxx.csv.txt}.

[0121] In one embodiment of the present application, the compressed key-value files include multiple files, and the key-value verification files include multiple files.

[0122] That is, in the optional embodiment, the index files obtained by the snapshot party include multiple ones, so the snapshot party searches for the latest value element corresponding to the same key element in each index file, and generates a key value file corresponding to each index file based on the key-value pair matching the latest value element, so as to obtain multiple key value files. After that, each key value file is compressed, that is, multiple compressed key value files are obtained; after that, each compressed key value file is hashed to obtain hash data corresponding to each compressed key value file, and a key value verification file is generated based on the hash data corresponding to each key value file, so as to obtain multiple key value verification files.

[0123] For example, see Table 3, which is an example of multiple key-value files and key-value verification files.

[0124]

[0125] Table 3

[0126] As shown in Table 3, for the index file 000010.ldb, the latest value element corresponding to the same key element is searched from the 000010.ldb file, and the key value file 1_stateDBSavepointKey.csv is generated based on the key-value pair matching the latest value element. Then, the 1_stateDBSavepointKey.csv file is compressed to obtain the compressed key value file 1_stateDBSavepointKey.csv.7z, and the key value verification file 1_stateDBSavepointKey.csv.txt is generated based on the 1_stateDBSavepointKey.csv.7z file.

[0127] For the index file 000011.ldb, the latest value element corresponding to the same key element is searched from the 000011.ldb file, and the key value file 2_theightKey.csv is generated based on the key-value pair matching the latest value element. Then, the 2_theightKey.csv file is compressed to obtain the compressed key value file 2_theightKey.csv.7z, and the key value verification file 2_theightKey.csv.txt is generated based on the 2_theightKey.csv.7z file.

[0128] Accordingly, the process of generating the snapshot file based on the archive file, the compressed key-value file, and the key-value verification file in S603 may include:

[0129] For each compressed key-value file, the compressed key-value file and the key-value verification file corresponding to the compressed key-value file are associated through a preset file name definition strategy;

[0130] The archive file, the associated key-value file and the key-value verification file are combined to obtain a snapshot file.

[0131] That is, in the optional embodiment, the snapshot party obtains multiple compressed key-value files and key-value verification files corresponding to the multiple compressed key-value files, and then associates each compressed key-value file with the key-value verification file corresponding to each compressed key-value file through a preset file name definition strategy, and then combines the archive file, the associated key-value file and the key-value verification file to obtain a snapshot file.

[0132] Among them, in an optional embodiment, the process of associating the compressed key-value file and the key-value verification file corresponding to the compressed key-value file through a preset file name definition strategy may include: setting the same file name prefix for the compressed key-value file and the key-value verification file corresponding to the compressed key-value file.

[0133] For example, referring to Table 3, the compressed key-value file 1_stateDBSavepointKey.csv.7z and its corresponding key-value verification file 1_stateDBSavepointKey.csv.txt both have a file name prefix of 1_stateDBSavepointKey; the compressed key-value file 2_theightKey.csv.7z and its corresponding key-value verification file 2_theightKey.csv.txt both have a file name prefix of 2_theightKey.

[0134] Among them, the preset file name definition strategy in the optional embodiment includes setting the file name of the key-value file to the starting value of the key element contained in the key-value file.

[0135] For example, assume that the preset file name definition strategy can be x_yyyy.csv, where x represents the number of the key-value file, and yyyy represents the first K in the key-value file; continuing with the previous example, for the 1_stateDBSavepointKey.csv.7z file, it is the first key-value file, and the first K in the key-value file is stateDBSavepointKey, and for the 2_theightKey.csv.7z file, it is the second key-value file, and the first K in the key-value file is theightKey.

[0136] Among them, in an optional embodiment, the snapshot file includes an archive file, and an associated key-value file and a key-value verification file; for example, following the above example, F represents the snapshot file, and assuming that the xxx.fdb file is specifically 0000000000000000000001.fdb and 0000000000000000000006.fdb, then F={00000000000000000000001.fdb, [1_stateDBSavepointKey.csv.7z, 1_stateDBSavepointKey.csv.txt], 000000000000000000006.fdb, [2_theightKey.csv.7z, 2_theightKey.csv.txt]}.

[0137] In one embodiment of the present application, after the process of combining the archive file, the associated key-value file and the key-value verification file to obtain the snapshot file, the following may also be included:

[0138] If it is detected that there is a target key-value file that is not associated with the key-value verification file in the snapshot file, regenerate the target key-value file based on the detected target key-value file, and regenerate the target key-value verification file corresponding to the newly generated target key-value file;

[0139] The newly generated target key-value file is associated with the target key-value verification file, and the detected target key-value file is replaced with the associated target key-value file and the target key-value verification file.

[0140] That is, in an optional embodiment, after the snapshot party obtains the snapshot file, it can detect the pairing of the key-value file and the key-value verification file in the snapshot file. If it is detected that the key-value file and the key-value verification file are not paired, a paired key-value file and key-value verification file can be regenerated. If it is detected that the key-value file and the key-value verification file are paired, no processing is performed.

[0141] In the optional embodiment, the target key-value file refers to a key-value file in the snapshot file that is not associated with the key-value verification file.

[0142] For example, following the above example, F={000000000000000000001.fdb,[1_stateDBSavepointKey.csv.7z,1_stateDBSavepointKey.csv.txt],00000000000000000006.fdb,[2_theightKey.csv.7z,2_theightKey.csv.txt]}, but due to network or other reasons, F={0000000000000 00000001.fdb, [1_stateDBSavepointKey.csv.7z], 000000000000000000006.fdb, [2_theightKey.csv.7z, 2_theightKey.csv.txt]}. At this time, it can be detected that the 1_stateDBSavepointKey.csv.7z file has no associated key value verification file, so the 1_stateDBSavepointKey.csv.7z file is the target key value file.

[0143] Afterwards, the 1_stateDBSavepointKey.csv.7z file may be regenerated based on the detected 1_stateDBSavepointKey.csv.7z file, and the target key value verification file 1_stateDBSavepointKey.csv.txt corresponding to the newly generated 1_stateDBSavepointKey.csv.7z file may be regenerated.

[0144] After that, the newly generated 1_stateDBSavepointKey.csv.7z file is associated with the newly generated 1_stateDBSavepointKey.csv.txt, and the detected 1_stateDBSavepointKey.csv.7z file is replaced with the associated 1_stateDBSavepointKey.csv.7z file and 1_stateDBSavepointKey.csv.txt.

[0145] In one embodiment of the present application, the process of regenerating the target key-value file based on the detected target key-value file may include:

[0146] Determine a key element interval corresponding to the target key-value file to be newly generated based on the file name corresponding to the detected target key-value file and the file names corresponding to key-value files adjacent to the detected target key-value file;

[0147] Regenerate the target key-value file based on the key element range.

[0148] That is, in an optional embodiment, the snapshot party can determine the key element interval corresponding to the target key value file to be newly generated based on the file name corresponding to the detected target key value file and the file name corresponding to the key value file adjacent to the detected target key value file, and then regenerate the target key value file based on the key element interval.

[0149] Among them, in an optional embodiment, the process of determining the key element interval corresponding to the target key-value file to be newly generated based on the file name corresponding to the detected target key-value file and the file name corresponding to the key value file adjacent to the detected target key-value file may include: determining the starting value of the key element interval based on the file name corresponding to the detected target key-value file, and determining the ending value of the key element interval based on the file name corresponding to the next key value file adjacent to the detected target key-value file, and determining the key element interval corresponding to the target key-value file to be newly generated based on the starting value and the ending value.

[0150] For example, following the previous example, based on the detected file name 1_stateDBSavepointKey.csv.7z of the file 1_stateDBSavepointKey.csv.7z, the starting value stateDBSavepointKey of the key element interval can be determined, and based on the file name 2_theightKey.csv.7z of the file 2_theightKey.csv.7z, the ending value theightKey of the key element interval can be determined, so that the key element interval can be determined as [stateDBSavepointKey, theightKey) based on the starting value stateDBSavepointKey and the ending value theightKey.

[0151] In one embodiment of the present application, after combining the archive file, the associated key-value file and the key-value verification file to obtain the snapshot file, the following may also be included:

[0152] If it is detected that there is a target key-value verification file that is not associated with the key-value file in the snapshot file, regenerate the target key-value file based on the detected target key-value verification file, and regenerate the target key-value verification file corresponding to the newly generated target key-value file;

[0153] The newly generated target key-value file is associated with the target key-value verification file, and the detected target key-value verification file is replaced with the associated target key-value file and target key-value verification file.

[0154] That is, in an optional embodiment, after the snapshot party obtains the snapshot file, it can detect the pairing of the key-value file and the key-value verification file in the snapshot file. If it is detected that the key-value file and the key-value verification file are not paired, a paired key-value file and key-value verification file can be regenerated. If it is detected that the key-value file and the key-value verification file are paired, no processing is performed.

[0155] Among them, in the optional embodiment, the target key-value verification file refers to a key-value verification file that is not associated with the key-value file in the snapshot file.

[0156] For example, following the above example, F={000000000000000000001.fdb,[1_stateDBSavepointKey.csv.7z,1_stateDBSavepointKey.csv.txt],00000000000000000006.fdb,[2_theightKey.csv.7z,2_theightKey.csv.txt]}, but due to network or other reasons, F={00000000000000 0000001.fdb, [1_stateDBSavepointKey.csv.txt], 000000000000000000006.fdb, [2_theightKey.csv.7z, 2_theightKey.csv.txt]}, at this time it can be detected that the 1_stateDBSavepointKey.csv.txt file has no associated key value file, then the 1_stateDBSavepointKey.csv.txt file is the target key value verification file.

[0157] Afterwards, the 1_stateDBSavepointKey.csv.7z file may be regenerated based on the detected 1_stateDBSavepointKey.csv.txt file, and the target key value verification file 1_stateDBSavepointKey.csv.txt corresponding to the newly generated 1_stateDBSavepointKey.csv.7z file may be regenerated.

[0158] Afterwards, the newly generated 1_stateDBSavepointKey.csv.7z file is associated with the newly generated 1_stateDBSavepointKey.csv.txt, and the detected 1_stateDBSavepointKey.csv.txt file is replaced with the associated 1_stateDBSavepointKey.csv.7z file and 1_stateDBSavepointKey.csv.txt.

[0159] In one embodiment of the present application, the process of regenerating the target key-value file based on the detected target key-value verification file may include:

[0160] Based on the file name corresponding to the detected target key-value verification file, determine the file name corresponding to the target key-value file to be newly generated;

[0161] Determine a key element interval corresponding to the target key-value file to be newly generated based on the file name corresponding to the target key-value file to be newly generated and the file names corresponding to key-value files adjacent to the target key-value file to be newly generated;

[0162] Regenerate the target key-value file based on the key element range.

[0163] That is, in an optional embodiment, the snapshot party can determine the file name corresponding to the target key-value file to be newly generated based on the file name corresponding to the detected target key-value verification file, and then determine the key element interval corresponding to the target key-value file to be newly generated based on the file name corresponding to the target key-value file to be newly generated and the file names corresponding to the key value files adjacent to the target key-value file to be newly generated, and then regenerate the target key-value file based on the key element interval.

[0164] Among them, because the key-value file and the key-value verification file are set with the same file name prefix; therefore, in an optional embodiment, the file name corresponding to the target key-value file to be newly generated can be determined based on the file name corresponding to the detected target key-value verification file.

[0165] For example, following the above example, based on the detected file name 1_stateDBSavepointKey.csv.txt of the file 1_stateDBSavepointKey.csv.txt, it can be determined that the file name corresponding to the target key-value file to be newly generated is 1_stateDBSavepointKey.csv.7z.

[0166] Among them, in an optional embodiment, the process of determining the key element interval corresponding to the target key-value file to be newly generated based on the file name corresponding to the target key-value file to be newly generated and the file name corresponding to the key value file adjacent to the target key-value file to be newly generated may include: determining the starting value of the key element interval based on the file name corresponding to the target key-value file to be newly generated, and determining the ending value of the key element interval based on the file name corresponding to the next key value file adjacent to the target key-value file to be newly generated, and determining the key element interval corresponding to the target key-value file to be newly generated based on the starting value and the ending value.

[0167] For example, following the previous example, based on the file name 1_stateDBSavepointKey.csv.7z of the newly generated 1_stateDBSavepointKey.csv.7z file, the starting value stateDBSavepointKey of the key element interval can be determined, and based on the file name 2_theightKey.csv.7z of the 2_theightKey.csv.7z file, the ending value theightKey of the key element interval can be determined, so that the key element interval can be determined as [stateDBSavepointKey, theightKey) based on the starting value stateDBSavepointKey and the ending value theightKey.

[0168] It should be noted that Figure 6 For detailed description of S301 to S302 and S304, please refer to Figure 3 S301 to S302 and S304 shown are not described in detail here.

[0169] In the embodiment of the present application, the snapshot file generated by the snapshot party includes a compressed key-value file. Since the key-value file is compressed, the data volume of the snapshot file is small and the storage space occupied is small. At the same time, the snapshot file generated by the snapshot party includes a key-value verification file for verifying the compressed key-value file. Therefore, the compressed key-value file can be verified separately, which ensures the security of the key-value file and is suitable for a wide range of scenarios.

[0170] In one embodiment of the present application, another blockchain-based data processing method is provided, which can be executed by the snapshot party 201. Figure 7 As shown, the blockchain-based data processing method may include S701 to S703, S301 to S302, and S304.

[0171] S701 to S703 are described in detail as follows:

[0172] S701, compress the archive file, and perform a hash operation on the compressed archive file to obtain hash data corresponding to the compressed archive file.

[0173] In the embodiment of the present application, the snapshot party obtains the archive file, and then can compress the archive file, and perform a hash operation on the compressed archive file to obtain hash data corresponding to the compressed archive file.

[0174] For example, following the previous example, the archive file is represented by the xxx.fdb file, where the 7z compression algorithm can be used to compress the xxx.fdb file to obtain the xxx.fdb.7z file, and then the hash algorithm is used to perform a hash operation on the xxx.fdb.7z file, i.e. Hash(xxx.fdb.7z file), to obtain the hash data hash_xxx.fdb.7z of the xxx.fdb.7z file.

[0175] S702: Generate an archive verification file for verifying the archive file based on hash data corresponding to the archive file.

[0176] In the embodiment of the present application, the snapshot party obtains the hash data corresponding to the archive file, and then can generate an archive verification file based on the hash data corresponding to the archive file.

[0177] In the embodiment of the present application, the archive verification file refers to a file generated by using the characteristic information / attribute information of the archive file to verify the archive file, and the characteristic information / attribute information of the archive file includes but is not limited to the hash data of the archive file. Among them, the archive verification file can be represented by the xxx.fdb.txt file.

[0178] For example, following the above example, an archive verification file xxx.fdb.txt is generated based on hash_xxx.fdb.7z.

[0179] S703, generating a snapshot file based on the key-value file, the compressed archive file, and the archive verification file.

[0180] In the embodiment of the present application, the snapshot party obtains a compressed archive file and an archive verification file, and then generates a snapshot file based on the key-value file, the compressed archive file, and the archive verification file.

[0181] In the embodiment of the present application, the snapshot file includes a key-value file, a compressed archive file, and an archive verification file; for example, continuing the above example, F represents the snapshot file, then F = {xxx.csv, xxx.fdb.7z, xxx.fdb.txt}.

[0182] In one embodiment of the present application, the compressed archive file includes multiple files, and the archive verification file includes multiple files.

[0183] That is, in an optional embodiment, the archive files obtained by the snapshot party include multiple files. Therefore, the snapshot party compresses each archive file, that is, obtains multiple compressed archive files; then, performs a hash operation on each compressed archive file to obtain hash data corresponding to each compressed archive file, and generates an archive verification file based on the hash data corresponding to each archive file, that is, obtains multiple archive verification files.

[0184] For example, see Table 4, which is an example of multiple archive files and archive verification files.

[0185]

[0186] Table 4

[0187] As shown in Table 4, for the archive file 000000000000000000001.fdb, the 00000000000000000000001.fdb file is compressed to obtain the compressed archive file 0000000000000000000001.fdb.7z, and the archive verification file 000000000000000000001.fdb.txt is generated based on the 000000000000000000001.fdb.7z file.

[0188] For the archive file 000000000000000000006.fdb, the 0000000000000000000006.fdb file is compressed to obtain the compressed archive file 000000000000000000006.fdb.7z, and the archive verification file 000000000000000000006.fdb.txt is generated based on the 000000000000000000006.fdb.7z file.

[0189] Accordingly, the process of generating the snapshot file based on the key-value file, the compressed archive file, and the archive verification file in S703 may include:

[0190] For each compressed archive file, the compressed archive file and the archive verification file corresponding to the compressed archive file are associated through a preset file name definition strategy;

[0191] The key-value file, the associated archive file and the archive verification file are combined to obtain a snapshot file.

[0192] That is, in an optional embodiment, the snapshot party obtains multiple compressed archive files and archive verification files corresponding to the multiple compressed archive files, and then associates each compressed archive file with the archive verification file corresponding to each compressed archive file through a preset file name definition strategy, and then combines the key value file, and the associated archive file and archive verification file to obtain a snapshot file.

[0193] Among them, in an optional embodiment, the process of associating the compressed archive file and the archive verification file corresponding to the compressed archive file through a preset file name definition strategy may include: setting the same file name prefix for the compressed archive file and the archive verification file corresponding to the compressed archive file.

[0194] For example, referring to Table 4, the compressed archive file 000000000000000000001.fdb.7z and its corresponding archive verification file 000000000000000000001.fdb.txt both have a file name prefix of 000000000000000000001; the compressed archive file 000000000000000000006.fdb.7z and its corresponding archive verification file 0000000000000000000006.fdb.txt both have a file name prefix of 00000000000000000006.

[0195] Among them, the preset file name definition strategy in the optional embodiment includes setting the file name of the archive file to the starting value of the block contained in the archive file.

[0196] For example, following the previous example, for the 0000000000000000000001.fdb.7z file, the height of the first block in the archive file is 1, and for the 0000000000000000000006.fdb.7z file, the height of the first block in the archive file is 6.

[0197] Among them, in an optional embodiment, the snapshot file includes a key-value file, and an associated archive file and an archive verification file; for example, following the above example, F represents the snapshot file, and assuming that the xxx.csv files are specifically 1_stateDBSavepointKey.csv and 2_theightKey.csv, then F={1_stateDBSavepointKey.csv, [000000000000000000001.fdb.7z, 000000000000000000001.fdb.txt], 2_theightKey.csv, [000000000000000000006.fdb.7z, 00000000000000000006.fdb.txt]}.

[0198] In one embodiment of the present application, after the process of combining the key value file, the associated archive file and the archive verification file to obtain the snapshot file, the following may also be included:

[0199] If it is detected that there is a target archive file in the snapshot file that is not associated with the archive verification file, regenerate the target archive file based on the detected target archive file, and regenerate the target archive verification file corresponding to the newly generated target archive file;

[0200] The newly generated target archive file is associated with the target archive verification file, and the detected target archive file is replaced with the associated target archive file and target archive verification file.

[0201] That is, in an optional embodiment, after the snapshot party obtains the snapshot file, it can check the pairing of the archive file and the archive verification file in the snapshot file. If it is detected that the archive file and the archive verification file are not paired, a paired archive file and archive verification file can be regenerated. If it is detected that the archive file and the archive verification file are paired, no processing is performed.

[0202] In the optional embodiment, the target archive file refers to an archive file in the snapshot file that is not associated with the archive verification file.

[0203] For example, following the above example, F = {1_stateDBSavepointKey.csv, [000000000000000000001.fdb.7z, 000000000000000000001.fdb.txt], 2_theightKey.csv, [00000000000000000006.fdb.7z, 00000000000000000006.fdb.txt]}, but due to network or other reasons, F = {1_stateDBSavepointKey.csv, [00000000000000000000006.fdb.txt]} pointKey.csv, [000000000000000000001.fdb.7z], 2_theightKey.csv, [00000000000000000006.fdb.7z, 000000000000000000006.fdb.txt]}, at this time it can be detected that the 0000000000000000000001.fdb.7z file has no associated archive verification file, then the 0000000000000000000001.fdb.7z file is the target archive file.

[0204] Afterwards, the 0000000000000000000001.fdb.7z file may be regenerated based on the detected 00000000000000000001.fdb.7z file, and the target archive verification file 0000000000000000000001.fdb.txt corresponding to the newly generated 0000000000000000000001.fdb.7z file may be regenerated.

[0205] Afterwards, the newly generated 0000000000000000000001.fdb.7z file is associated with the newly generated 0000000000000000000001.fdb.txt, and the detected 0000000000000000000001.fdb.7z file is replaced with the associated 00000000000000000000001.fdb.7z file and 00000000000000000000001.fdb.txt.

[0206] In one embodiment of the present application, the process of regenerating the target archive file based on the detected target archive file may include:

[0207] Based on the file name corresponding to the detected target archive file and the file name corresponding to the archive file adjacent to the detected target archive file, determine the block interval corresponding to the target archive file to be newly generated;

[0208] Regenerate the target archive based on the chunk range.

[0209] That is, in an optional embodiment, the snapshot party can determine the block interval corresponding to the target archive file to be newly generated based on the file name corresponding to the detected target archive file and the file name corresponding to the archive file adjacent to the detected target archive file, and then regenerate the target archive file based on the block interval.

[0210] Among them, in an optional embodiment, the process of determining the block interval corresponding to the target archive file to be newly generated based on the file name corresponding to the detected target archive file and the file name corresponding to the archive file adjacent to the detected target archive file may include: determining the starting value of the block interval based on the file name corresponding to the detected target archive file, and determining the ending value of the block interval based on the file name corresponding to the next archive file adjacent to the detected target archive file, and determining the block interval corresponding to the target archive file to be newly generated based on the starting value and the ending value.

[0211] For example, following the previous example, based on the detected file name 0000000000000000000001.fdb.7z file 000000000000000000001.fdb.7z, the starting value 1 of the block interval can be determined, and based on the file name 00000000000000000006.fdb.7z file 00000000000000000006.fdb.7z, the ending value 6 of the block interval can be determined, so that the block interval can be determined as [1, 6) based on the starting value 1 and the ending value 6.

[0212] In one embodiment of the present application, after combining the archive file, and the associated archive file and archive verification file to obtain the snapshot file, the following may also be included:

[0213] If it is detected that there is a target archive verification file not associated with the archive file in the snapshot file, regenerate the target archive file based on the detected target archive verification file, and regenerate the target archive verification file corresponding to the newly generated target archive file;

[0214] The newly generated target archive file is associated with the target archive verification file, and the detected target archive verification file is replaced with the associated target archive file and target archive verification file.

[0215] That is, in an optional embodiment, after the snapshot party obtains the snapshot file, it can check the pairing of the archive file and the archive verification file in the snapshot file. If it is detected that the archive file and the archive verification file are not paired, a paired archive file and archive verification file can be regenerated. If it is detected that the archive file and the archive verification file are paired, no processing is performed.

[0216] In the optional embodiment, the target archive verification file refers to an archive verification file in the snapshot file that is not associated with the archive file.

[0217] For example, following the above example, F = {1_stateDBSavepointKey.csv, [000000000000000000001.fdb.7z, 000000000000000000001.fdb.txt], 2_theightKey.csv, [00000000000000000006.fdb.7z, 00000000000000000006.fdb.txt]}, but due to network or other reasons, F = {1_stateDBSavepointKey.csv, [00000000000000000000006.fdb.txt]} ointKey.csv,[000000000000000000001.fdb.txt],2_theightKey.csv,[00000000000000000006.fdb.7z,00000000000000000006.fdb.txt]}, at this time it can be detected that the 000000000000000000001.fdb.txt file has no associated archive file, then the 0000000000000000000001.fdb.txt file is the target archive verification file.

[0218] Afterwards, the 000000000000000000001.fdb.txt file may be regenerated based on the detected 00000000000000000001.fdb.7z file, and the target archive verification file 0000000000000000000001.fdb.txt corresponding to the newly generated 000000000000000000001.fdb.7z file may be regenerated.

[0219] Afterwards, the newly generated 0000000000000000000001.fdb.7z file is associated with the newly generated 0000000000000000000001.fdb.txt, and the detected 0000000000000000000001.fdb.txt file is replaced with the associated 00000000000000000000001.fdb.7z file and 0000000000000000000001.fdb.txt.

[0220] In one embodiment of the present application, the process of regenerating the target archive file based on the detected target archive verification file may include:

[0221] Based on the file name corresponding to the detected target archive verification file, determine the file name corresponding to the target archive file to be newly generated;

[0222] Determine the block interval corresponding to the target archive file to be newly generated based on the file name corresponding to the target archive file to be newly generated and the file name corresponding to the archive file adjacent to the target archive file to be newly generated;

[0223] Regenerate the target archive based on the chunk range.

[0224] That is, in an optional embodiment, the snapshot party can determine the file name corresponding to the target archive file to be newly generated based on the file name corresponding to the detected target archive verification file, and then determine the block interval corresponding to the target archive file to be newly generated based on the file name corresponding to the target archive file to be newly generated and the file names corresponding to archive files adjacent to the target archive file to be newly generated, and then regenerate the target archive file based on the block interval.

[0225] Among them, because the archive file and the archive verification file are set with the same file name prefix; therefore, in an optional embodiment, the file name corresponding to the target archive file to be newly generated can be determined based on the file name corresponding to the detected target archive verification file.

[0226] For example, following the aforementioned example, based on the detected file name 0000000000000000000001.fdb.txt of the file 000000000000000000001.fdb.txt, it can be determined that the file name corresponding to the target archive file to be newly generated is 0000000000000000000001.fdb.7z.

[0227] Among them, in an optional embodiment, the process of determining the block interval corresponding to the target archive file to be newly generated based on the file name corresponding to the target archive file to be newly generated and the file name corresponding to the archive file adjacent to the target archive file to be newly generated may include: determining the starting value of the block interval based on the file name corresponding to the target archive file to be newly generated, and determining the ending value of the block interval based on the file name corresponding to the next archive file adjacent to the target archive file to be newly generated, and determining the block interval corresponding to the target archive file to be newly generated based on the starting value and the ending value.

[0228] For example, following the previous example, based on the file name 0000000000000000000001.fdb.7z of the newly generated 000000000000000000001.fdb.7z file, the starting value 1 of the block interval can be determined, and based on the file name 00000000000000000006.fdb.7z file, the ending value 6 of the block interval can be determined, so that the block interval can be determined as [1, 6) based on the starting value 1 and the ending value 6.

[0229] It should be noted that Figure 7 For detailed description of S301 to S302 and S304, please refer to Figure 3 S301 to S302 and S304 shown are not described in detail here.

[0230] In the embodiment of the present application, the snapshot file generated by the snapshot party includes a compressed archive file. Since the archive file is compressed, the data volume of the snapshot file is small and the storage space occupied is small. At the same time, the snapshot file generated by the snapshot party includes an archive verification file for verifying the compressed archive file. Therefore, the compressed archive file can be verified separately, thereby ensuring the security of the archive file and being suitable for a wide range of scenarios.

[0231] It should be clear that the generation of snapshot files based on key-value files, compressed archive files, and archive verification files in the embodiments of the present application can be combined with the generation of snapshot files based on archive files, compressed key-value files, and key-value verification files in the aforementioned embodiments; that is, in other embodiments, snapshot files can also be generated based on compressed archive files, archive verification files, compressed key-value files, and key-value verification files. For the specific implementation process, please refer to the aforementioned introduction, which will not be repeated here.

[0232] Among them, in other embodiments, the snapshot file generated by the snapshot party includes a compressed archive file and a compressed key-value file. Since both the archive file and the compressed file are compressed, the data volume of the snapshot file is small and the storage space occupied is small. At the same time, the snapshot file generated by the snapshot party includes an archive verification file for verifying the compressed archive file, so the compressed archive file can be verified separately, ensuring the security of the archive file, and the snapshot file generated by the snapshot party includes a key-value verification file for verifying the compressed key-value file, so the compressed key-value file can be verified separately, ensuring the security of the key-value file, which is suitable for a wide range of scenarios.

[0233] In one embodiment of the present application, another blockchain-based data processing method is provided, which can be executed by the snapshot party 201. Figure 8 As shown, the blockchain-based data processing method may include S801, S301, S303 to S304.

[0234] In the embodiment of the present application, the key-value pairs contained in the index file are stored in multiple shard areas.

[0235] S801 is described in detail as follows:

[0236] S801, searching for the latest value element corresponding to the same key element in each shard area in parallel, and generating a key-value file corresponding to the index file based on the key-value pair matching the latest value element.

[0237] In the embodiment of the present application, the snapshot party can search for the latest value element corresponding to the same key element from each shard area in parallel, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element.

[0238] Exemplarily, the snapshot party can start multiple threads, where the number of threads matches the number of shard regions, that is, one thread corresponds to one shard region, and the threads are used to search for the latest value element corresponding to the same key element from the corresponding shard region, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element.

[0239] For example, suppose there are three shard regions, namely region1, region2, and region3. Thread 1 is used to search for the latest value element corresponding to the same key element from region1, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element. At the same time, thread 2 is used to search for the latest value element corresponding to the same key element from region2, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element. Thread 3 is used to search for the latest value element corresponding to the same key element from region3, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element.

[0240] It should be noted that Figure 8 For detailed description of S301, S303 to S304, please refer to Figure 3 S301, S303 to S304 shown are not described in detail here.

[0241] In the embodiment of the present application, the snapshot party generates multiple key-value files in parallel, which improves the efficiency of generating key-value files, thereby improving the efficiency of data snapshots.

[0242] In one embodiment of the present application, another blockchain-based data processing method is provided, which can be executed by the snapshot party 201. Fig. 9 As shown, the blockchain-based data processing method may further include S901 to S902 after S302.

[0243] In the embodiment of the present application, the index files include multiple index files, that is, the index files obtained by the snapshot party include multiple index files. Therefore, the snapshot party searches for the latest value element corresponding to the same key element in each index file, and generates a key-value file corresponding to each index file based on the key-value pair matching the latest value element, so as to obtain multiple key-value files.

[0244] S901 to S902 are described in detail as follows:

[0245] S901: If it is detected that an interruption occurs in the process of generating a key-value file corresponding to an index file, a starting value corresponding to a key element to be searched is determined based on the file name corresponding to the generated key-value file.

[0246] It is understandable that during the process of the snapshot party generating the key-value file corresponding to the index file, an interruption may occur due to abnormal conditions (such as a crash, power outage, etc.); therefore, in the embodiment of the present application, if the snapshot party detects that an interruption occurs during the process of generating the key-value file corresponding to the index file, the starting value corresponding to the key element to be searched can be determined based on the file name corresponding to the generated key-value file.

[0247] As described above, the file name corresponding to the key-value file is set to the starting value of the key element contained in the key-value file; therefore, in an embodiment of the present application, the starting value corresponding to the key element to be searched can be determined based on the file name corresponding to the generated key-value file.

[0248] In one embodiment of the present application, the process of determining the starting value corresponding to the key element to be searched based on the file name corresponding to the generated key-value file in S901 may include: searching for the latest generated key-value file from multiple generated key-value files, and determining the starting value corresponding to the key element to be searched based on the file name corresponding to the latest generated key-value file.

[0249] Among them, the newly generated key value file in the optional embodiment refers to the key value file whose generation time is closest to the current time among multiple generated key value files. In short, it is the key value file whose generation time has the smallest difference with the current time among multiple generated key value files.

[0250] S902, searching for the latest value element corresponding to the starting value starting from the starting value, and generating a key-value file corresponding to the index file based on the key-value pair matching the latest value element corresponding to the starting value, until multiple key-value files corresponding to the index files are generated.

[0251] In the embodiment of the present application, the snapshot party determines the starting value corresponding to the key element to be searched, and then starts to search for the latest value element corresponding to the starting value from the starting value, and generates a key value file corresponding to the index file based on the key-value pair matching the latest value element corresponding to the starting value, until key value files corresponding to multiple index files are generated.

[0252] For example, if the highest block height of the blockchain where the snapshot party is located is 100, and there are 20 index files generated, then the snapshot party should complete the snapshot of 20 index files. Assume that when the snapshot party snapshots to the 10th index file, the snapshot is interrupted due to a crash. Then the snapshot party can find the 9th key-value file from the 10 key-value files that have been generated, and based on the file name corresponding to the 9th key-value file, determine the starting value corresponding to the key element to be found, and then start from the starting value to find the latest value element corresponding to the starting value, and generate the 10th key-value file based on the key-value pair that matches the latest value element corresponding to the starting value, until 20 key-value files are generated.

[0253] It should be noted that Fig. 9 For detailed description of S301 to S304, please refer to Figure 4 S301 to S304 shown are not described in detail here.

[0254] In the embodiment of the present application, after an interruption occurs during the key-value file generation process, the snapshot party can continue to generate the key-value file from the interruption point without having to start the key-value file generation from scratch, which greatly improves the key-value file generation efficiency and thus improves the efficiency of the data snapshot.

[0255] It should be noted that Figures 2 to 9 The embodiment shown is explained from the perspective of the snapshot side. Figures 10 to 13 The implementation details of the technical solution of the embodiment of the present application are described in detail from the perspective of the importer:

[0256] See also Fig.10 , Fig.10 is a flowchart of a data processing method based on blockchain shown in an embodiment of the present application. The data processing method based on blockchain can be executed by the importer 202. Fig.10 As shown, the blockchain-based data processing method includes at least S1001 to S1003, which are described in detail as follows:

[0257] S1001, if a data import request is received, obtain the associated stored snapshot file and snapshot verification file; wherein the snapshot file includes a key-value file and an archive file, and the key-value file is generated based on the key-value pairs that match the latest value elements corresponding to the same key element.

[0258] In an embodiment of the present application, when the requesting party has a demand for data import, a data import request can be issued, and then the data import request can be sent to the importing party; accordingly, the importing party receives the data import request sent by the requesting party, and can obtain the associated stored snapshot file and snapshot verification file, wherein the snapshot file includes a key-value file and an archive file, and the key-value file is generated based on a key-value pair that matches the latest value element corresponding to the same key element. For details, please see the above introduction, which will not be repeated here.

[0259] In one embodiment of the present application, the process of obtaining the associated stored snapshot file and snapshot verification file in S1001 may include:

[0260] Obtaining a snapshot file from the snapshot party, and decompressing the snapshot file to obtain connection information for establishing communication with the snapshot party;

[0261] Obtain the snapshot verification file corresponding to the snapshot file from the snapshot party based on the connection information.

[0262] That is, in an optional embodiment, the snapshot party stores the compressed snapshot file in a data warehouse, and stores the snapshot verification file corresponding to the snapshot file in a local storage area. The importing party can first obtain the snapshot file from the data warehouse, and decompress the snapshot file to obtain connection information for establishing communication with the snapshot party, and then establish communication with the snapshot party based on the connection information to obtain the snapshot verification file corresponding to the snapshot file from the local storage area of ​​the snapshot party based on the communication link.

[0263] In an optional embodiment, the connection information may also be stored in a separate file; for example, it may be stored in a connection file. Accordingly, the connection file may be obtained from a data warehouse, and the connection file may be decompressed to obtain the connection information.

[0264] S1002: If the snapshot file is verified based on the snapshot verification file, the key-value file and the archive file are obtained from the snapshot file.

[0265] In the embodiment of the present application, the importer obtains the snapshot file and the snapshot verification file that are stored in association, and then can verify the snapshot file based on the snapshot verification file, and perform corresponding operations based on the verification result.

[0266] The process of verifying the snapshot file based on the snapshot verification file in the embodiment of the present application may include:

[0267] Performing a hash operation on the acquired snapshot file to generate hash data corresponding to the acquired snapshot file;

[0268] If the hash data corresponding to the generated snapshot file matches the hash data in the snapshot verification file, it is determined that the verification of the snapshot file passes;

[0269] If the hash data corresponding to the generated snapshot file does not match the hash data in the snapshot verification file, it is determined that the verification of the snapshot file has failed.

[0270] The process of performing corresponding operations based on the verification result in the embodiment of the present application may include:

[0271] In case 1, if the snapshot file passes verification, the key-value file and the archive file can be obtained from the snapshot file.

[0272] It can be understood that when the snapshot file is verified, it indicates that the snapshot file is highly secure, and thus the snapshot file can be trusted. At this time, the key-value file and the archive file can be obtained from the snapshot file.

[0273] In case 2, if the snapshot file fails to be verified, no processing may be performed, or a result indicating that the snapshot file fails to be verified may be returned, or other operations may be performed (see the following description).

[0274] It is understandable that when the snapshot file fails to be verified, it indicates that the snapshot file has low security, and thus the snapshot file cannot be trusted. In this case, there is no need to obtain the key-value file and the archive file from the snapshot file.

[0275] S1003, import the key-value file and the archive file into the local storage area.

[0276] In the embodiment of the present application, the importing party obtains the key-value file and the archive file from the snapshot file, and then can import the key-value file and the archive file into the local storage area, thereby realizing a snapshot of the snapshot file and providing support for joining the blockchain.

[0277] In the embodiment of the present application, the snapshot file obtained by the importer has a small amount of data, which saves storage space and transmission time, improves the efficiency of data import in the blockchain, and can also use the snapshot verification file to verify the snapshot file, and import the data after the verification passes, thereby improving the security of data import in the blockchain.

[0278] In one embodiment of the present application, another blockchain-based data processing method is provided, which can be executed by the importer 202. Fig.11 As shown, the blockchain-based data processing method may further include S1101 to S1103 after S1001.

[0279] In the embodiment of the present application, the key-value files include multiple ones, that is, the snapshot file includes an archive file, multiple key-value files, and a key-value verification file corresponding to each key-value file.

[0280] S1101 to S1103 are described in detail as follows:

[0281] S1101: If the snapshot file fails to be verified based on the snapshot verification file, a key-value verification file corresponding to each key-value file is obtained from the snapshot file.

[0282] In the embodiment of the present application, if the importer fails to verify the snapshot file based on the snapshot verification file, the key value verification file corresponding to each key value file can be obtained from the snapshot file, that is, each key value file can be verified separately.

[0283] S1102: For each key-value file, verify the key-value file based on a key-value verification file corresponding to the key-value file.

[0284] In the embodiment of the present application, the importer obtains the key-value verification file corresponding to each key-value file, and then can verify each key-value file based on the key-value verification file corresponding to each key-value file, and perform corresponding operations based on the verification results.

[0285] In the embodiment of the present application, the process of verifying the key-value file based on the key-value verification file corresponding to the key-value file may include:

[0286] Performing a hash operation on the obtained key-value file to generate hash data corresponding to the obtained key-value file;

[0287] If the hash data corresponding to the generated key-value file matches the hash data in the key-value verification file, it is determined that the verification of the key-value file has passed;

[0288] If the hash data corresponding to the generated key-value file does not match the hash data in the key-value verification file, it is determined that the verification of the key-value file has failed.

[0289] The process of performing corresponding operations based on the verification result in the embodiment of the present application may include:

[0290] Case 1: If the key-value file passes the verification, the key-value file can be imported into the local storage area.

[0291] It can be understood that when the key-value file is verified, it indicates that the key-value file is highly secure, and thus the key-value file can be trusted. At this time, the key-value file can be imported into the local storage area.

[0292] Case 2: If the key-value file fails to be verified, no processing may be performed, or a result indicating that the key-value file has failed to be verified may be returned, or other operations may be performed (eg, re-obtaining the corresponding key-value file based on the key-value file that has failed to be verified).

[0293] It is understandable that when the key-value file fails to be verified, it indicates that the key-value file has low security, and therefore the key-value file cannot be trusted. In this case, there is no need to import the key-value file into the local storage area.

[0294] For example, following the previous example, F = {0000000000000000000001.fdb, [1_stateDBSavepointKey.csv.7z, 1_stateDBSavepointKey.csv.txt], 000000000000000000006.fdb, [2_theightKey.csv.7z, 2_theightKey.csv.txt], then the 1_stateDBSavepointKey.csv.7z file is verified based on the 1_stateDBSavepointKey.csv.txt file, and the 2_theightKey.csv.7z file is verified based on the 2_theightKey.csv.txt file.

[0295] S1103, importing the verified key-value file into the local storage area.

[0296] For example, continuing with the above example, assuming that the verification of the 1_stateDBSavepointKey.csv.7z file passes, and the verification of the 2_theightKey.csv.7z file fails, the 1_stateDBSavepointKey.csv.7z file is imported into the local storage area.

[0297] It should be noted that Fig.11 For detailed description of S1001 to S1003, please refer to Fig.10 S1001 to S1003 shown are not described in detail here.

[0298] In the embodiment of the present application, if the snapshot file fails to be verified, the importing party further verifies the key-value file contained in the snapshot file, and imports the key-value file that has passed the verification into the local storage area, thereby realizing the import of partial data, which is highly flexible and suitable for a wide range of scenarios.

[0299] In one embodiment of the present application, another blockchain-based data processing method is provided, which can be executed by the importer 202. Fig.12 As shown, the blockchain-based data processing method may further include S1201 to S1203 after S1001.

[0300] In the embodiment of the present application, the archive file includes multiple files, that is, the snapshot file includes a key-value file, multiple archive files, and an archive verification file corresponding to each archive file.

[0301] S1201 to S1203 are described in detail as follows:

[0302] S1201: If the snapshot file fails to be verified based on the snapshot verification file, an archive verification file corresponding to each archive file is obtained from the snapshot file.

[0303] In the embodiment of the present application, if the importer fails to verify the snapshot file based on the snapshot verification file, the archive verification file corresponding to each archive file can be obtained from the snapshot file, that is, each archive file can be verified separately.

[0304] S1202: For each archive file, verify the archive file based on the archive verification file corresponding to the archive file.

[0305] In the embodiment of the present application, the importer obtains the archive verification file corresponding to each archive file, and then can verify each archive file based on the archive verification file corresponding to each archive file, and perform corresponding operations based on the verification results.

[0306] In the embodiment of the present application, the process of verifying the archive file based on the archive verification file corresponding to the archive file may include:

[0307] Performing a hash operation on the acquired archive file to generate hash data corresponding to the acquired archive file;

[0308] If the hash data corresponding to the generated archive file matches the hash data in the archive verification file, it is determined that the verification of the archive file passes;

[0309] If the hash data corresponding to the generated archive file does not match the hash data in the archive verification file, it is determined that the verification of the archive file has failed.

[0310] The process of performing corresponding operations based on the verification result in the embodiment of the present application may include:

[0311] Case 1: If the archive file passes verification, the archive file can be imported into the local storage area.

[0312] It can be understood that when the verification of the archive file passes, it indicates that the archive file has high security, so the archive file can be trusted, and the archive file can be imported into the local storage area at this time.

[0313] Case 2: If the archive file fails to be verified, no processing may be performed, or the archive file verification failure result may be returned, or other operations may be performed (eg, re-obtaining the corresponding archive file based on the archive file that failed the verification).

[0314] It is understandable that when the archive file fails to pass the verification, it indicates that the archive file has low security, so the archive file cannot be trusted. At this time, there is no need to import the archive file into the local storage area.

[0315] For example, following the above example, F = {1_stateDBSavepointKey.csv, [000000000000000000001.fdb.7z, 00000000000000000001.fdb.txt], 2_theightKey.csv, [000000000000000000006.fdb.7z, 00000000000 0000000006.fdb.txt]}, the 0000000000000000000001.fdb.txt file will be verified against the 0000000000000000000001.fdb.7z file, and the 0000000000000000000006.fdb.txt file will be verified against the 0000000000000000000006.fdb.7z file.

[0316] S1203, importing the archived files that have passed the verification into the local storage area.

[0317] For example, following the previous example, assuming that the verification of the 0000000000000000000001.fdb.7z file passes, and the verification of the 000000000000000000006.fdb.7z file fails, the 0000000000000000000001.fdb.7z file is imported into the local storage area.

[0318] It should be noted that Fig.12 For detailed description of S1001 to S1003, please refer to Fig.10 S1001 to S1003 shown are not described in detail here.

[0319] In the embodiment of the present application, if the snapshot file fails to be verified, the importing party further verifies the archive files contained in the snapshot file, and imports the archive files that have passed the verification into the local storage area, thereby realizing the import of partial data, which is highly flexible and suitable for a wide range of scenarios.

[0320] It should be clear that the verification of the archived files contained in the snapshot file in the embodiment of the present application and the verification of the key-value files contained in the snapshot file in the aforementioned embodiment can be combined; that is, in other embodiments, if the verification of the snapshot file based on the snapshot verification file fails, the archived files contained in the snapshot file can be verified, as well as the key-value files contained in the snapshot file. For the specific implementation process, please refer to the aforementioned introduction and will not be repeated here.

[0321] Among them, in other embodiments, the importing party verifies the archive files contained in the snapshot file, and imports the archive files that have passed the verification into the local storage area, thereby realizing the import of archive files with high flexibility. At the same time, the importing party verifies the key-value files contained in the snapshot file, and imports the key-value files that have passed the verification into the local storage area, thereby realizing the import of key-value files with high flexibility.

[0322] The following is a detailed description of a specific scenario of an embodiment of the present application:

[0323] First, see Fig.13 The overall architecture in the embodiment of the present application mainly includes:

[0324] Blockchain nodes: Blockchain Node 1 and Blockchain Node 2.

[0325] Among them, blockchain node 1 acts as a snapshot party, which can be a node in blockchain 1, including an RPC server (remote procedure call server) and an on-chain storage repository. The RPC server can be used to provide query services to the outside. The on-chain storage repository can be used to store on-chain data, which can be a local storage area of ​​blockchain node 1 and can interact with the storage warehouse; the storage warehouse can be used to store archive files and index files, and can also store snapshot data (such as snapshot files). The archive file is used to archive the chained blocks, and the index file is used to store the key-value pairs converted from the index data and status data related to multiple chained blocks.

[0326] Exemplarily, the file name of the xxx.fdb file (i.e., the archive file) is set to the result of converting the block height of the first block stored in the file according to the file name rule. This setting allows the user to see the xxx.fdb file where the first block in a batch of consecutive blocks is located through the file; for example, the xxx.fdb file contains 5 blocks with block heights of 0, 1, 2, 3, and 4, then the file name can be 00000000000000000000.fdb.

[0327] Exemplarily, a single xxx.fdb file storing blocks has a certain size limit, for example, each xxx.fdb is limited to 1024MB, regardless of how many blocks are stored; or a single xxx.fdb file storing blocks can also set the number of blocks stored, for example, each xxx.fdb file stores 100 blocks; in actual applications, the number of blocks in a single xxx.fdb file can be flexibly adjusted according to the specific application scenario.

[0328] For example, the archived file will have corresponding index information, wherein the index information can be specifically stored in kvdb (also called bfkv), and the structure of the index information is as follows:

[0329] type Index struct {

[0330] / / file name xxx.fdb file name

[0331] FileName string

[0332] / / Offset in file The offset of the block in the xxx.fdb file

[0333] Offset uint64`

[0334] / / The length of the data length block in the xxx.fdb file

[0335] ByteLen uint64

[0336] }

[0337] For example, the xxx.fdb file contains 10 blocks with block heights of 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. Suppose it is divided into two xxx.fdb files, then the inclusion relationship between xxx.fdb and blocks can be shown in Table 5:

[0338] file name Contains blocks Contains index 00000000000000000000.fdb 0、1、2、3、4 block, blockHeader, tx… 00000000000000000005.fdb 5、6、7、8、9 block, blockHeader, tx…

[0339] Table 5

[0340] It can be understood that a xxx.fdb file can be generated based on the included blocks, and a xxx.ldb file can be generated based on the included indexes.

[0341] See also Fig.14 , is a schematic diagram of the block in the xxx.fdb file. Fig.14 As shown, the serialized texts of multiple blocks constitute the xxx.fdb file, that is, a single xxx.fdb file includes the serialized texts of multiple blocks, wherein the serialized text of each block includes the serialized text of the block header, the serialized text of the transaction, and the serialized text of other objects in the block (such as read-write sets, etc.).

[0342] See also Fig.15 , a schematic diagram of archive file and index file storage. Fig.15As shown, the bfdb file directory includes multiple xxx.fdb files, and the xxx.fdb.END file is the xxx.fdb file being written. When the size of the xxx.fdb.END file being written reaches the limited size, it is renamed as the xxx.fdb file, and a new xxx.fdb.END file is generated at the same time. The Blockdb file directory includes multiple xxx.ldb files. Exemplarily, the folder corresponding to the Blockdb file directory uses leveldb (since rocksdb is an optimized version of leveldb, rocksdb can be used preferentially in the embodiment of the present application), where leveldb and rocksdb both belong to kvdb, and kvdb usually uses a log structure merge tree algorithm for key-value pair storage, and the log structure merge tree has the problem of write amplification. Therefore, this is optimized accordingly in the embodiment of the present application, and the details are described later.

[0343] Among them, blockchain node 2 is the importer, which can be the node to be added to blockchain 1, including RPC server and on-chain storage. RPC server can also be used to provide query services to the outside world ( Fig.13 The on-chain repository may be used to store on-chain data, which may be a local storage area of ​​the blockchain node 2 and may interact with the storage warehouse to obtain snapshot data from the storage warehouse for import.

[0344] Interactive tool: It can be used to interact with blockchain nodes. It encapsulates the interface calling method of blockchain nodes and can be an independently running small tool program, such as cmc or ChainMaker.sdk (a blockchain provided by ChainMaker for a certain enterprise).

[0345] like Fig.13 As shown, the blockchain-based data processing method may include S1301 to S1307, and S1301 to S1307 are described in detail as follows:

[0346] S1301, the interactive tool sends a data snapshot request to blockchain node 1.

[0347] Among them, the data snapshot request is used to instruct blockchain node 1 to perform a snapshot operation on local data.

[0348] S1302, blockchain node 1 performs a snapshot operation based on the data snapshot request.

[0349] Among them, blockchain node 1 needs to stop generating blocks and ensure that the data does not change during the snapshot operation. The snapshot file generated by the snapshot operation can be stored in the storage warehouse. The snapshot file can include the connection information (such as p2p or rpc) of blockchain node 1. Exemplarily, the user can store the snapshot file to another disk, or to a public network (such as a network disk) for sharing with other users.

[0350] Among them, the blockchain node 1 can also generate a snapshot verification file corresponding to the snapshot file (such as hash data of the snapshot file) so that the snapshot file can be verified based on the snapshot verification file. Exemplarily, the snapshot verification file can be stored in an on-chain repository or a credible system.

[0351] S1303, blockchain node 1 returns the snapshot operation execution result to the interactive tool.

[0352] The snapshot operation execution result includes whether the snapshot operation is executed successfully or failed.

[0353] S1304, the interactive tool sends a data import request to blockchain node 2.

[0354] The data import request is used to instruct blockchain node 2 to perform an import operation on the snapshot file.

[0355] S1305, blockchain node 2 obtains the snapshot file based on the data import request and verifies the snapshot file.

[0356] Among them, when blockchain node 2 obtains a snapshot file from the storage warehouse, it first generates a snapshot verification file corresponding to the snapshot file based on the obtained snapshot file, and at the same time reads the connection information of blockchain node 1 from the snapshot file, and uses the connection information of blockchain node 1 to establish communication with blockchain node 1, so as to obtain the snapshot verification file corresponding to the snapshot file from blockchain node 1 based on the communication link, and then compares the generated snapshot verification file with the obtained snapshot verification file. If the two are consistent, it is determined that the verification of the obtained snapshot file has passed. If the two are inconsistent, it is determined that the verification of the obtained snapshot file has failed.

[0357] S1306: If the verification of the snapshot file passes, blockchain node 2 performs the import operation.

[0358] Among them, blockchain node 2 can import the verified snapshot file into the on-chain repository.

[0359] For example, if blockchain node 1 stores the snapshot verification file in the on-chain repository, blockchain node 1 may not be online during the verification. At this time, blockchain node 2 can choose whether to import the unverified snapshot file. Because the security of the unverified snapshot file is unknown, it may not be added to blockchain 1 or be removed from blockchain 1 in the future.

[0360] Exemplarily, if blockchain node 1 stores the snapshot verification file in a credible system, the snapshot verification file can be obtained during verification, thereby completing the verification of the snapshot file.

[0361] S1307, blockchain node 2 returns the import operation execution result to the interactive tool.

[0362] The import operation execution result includes whether the import operation is executed successfully or failed.

[0363] At this point, the data snapshot and data import process is completed.

[0364] For ease of understanding, the data snapshot process (ie, S1301 to S1303) is described in detail below.

[0365] 1.1. Blockchain node 1 temporarily stops generating blocks after receiving a data snapshot request to ensure that the data remains unchanged during the snapshot period.

[0366] 1.2. Blockchain node 1 calculates the snapshot file height, where the snapshot file height is the latest block height on blockchain node 1.

[0367] 1.3. Blockchain node 1 creates a snapshot folder in the data directory, and the folder name is snapshot_xxx_yyy, where xxx represents the blockchain identifier, such as id, and yyy represents the latest block height corresponding to blockchain node 1 contained in the snapshot file (for example, snapshot_chain1_3000 means that the snapshot file of blockchain chain1 contains all data before and at the height of blockchain node 1 at 3000). For blockchain node 1 without multiple chains, chainid does not need to be paid attention to, and the folder name can be snapshot_xxx.

[0368] 1.4. Blockchain node 1 generates a bfdb snapshot folder based on the xxx.fdb file, and generates corresponding folders based on the xxx.ldb file, such as a blockdb snapshot folder, a statedb snapshot folder, a resultdb snapshot folder (also called a result database), and a historydb snapshot folder (also called a history database).

[0369] See also Fig.16, is a schematic diagram of the snapshot file structure. Fig.16 As shown, the snapshot_xxx_yyy data takes snapshot_chain1_3000 as an example, the bfdb snapshot folder includes multiple xxx.fdb.7z files (i.e. compressed archive files) and multiple xxx.fdb.txt files (i.e. archive verification files), the blockdb snapshot folder includes multiple xxx.csv files (i.e. key-value files) and multiple xxx.csv.txt files (i.e. key-value verification files), and the statedb snapshot folder includes multiple xxx.csv files (i.e. key-value files) and multiple xxx.csv.txt files (i.e. key-value verification files). Exemplarily, the blockdb snapshot folder may include multiple xxx.csv.7z files (i.e. compressed key-value files) and multiple xxx.csv.txt files (i.e. key-value verification files), and the statedb snapshot folder may include multiple xxx.csv.7z files (i.e. compressed key-value files) and multiple xxx.csv.txt files (i.e. key-value verification files).

[0370] Understandably, Fig.16 The xxx.fdb.txt file included in the bfdb snapshot folder is obtained by using a hash algorithm (such as the sha256 hash algorithm) to perform a hash operation on the xxx.fdb.7z file after the xxx.fdb.7z file is successfully saved to the disk, and the calculated hash data is stored in the xxx.fdb.7z file. The corresponding relationship can be seen in Table 4. At the same time, in order to ensure data security, the xxx.fdb.7z file and the xxx.fdb.txt file included in the bfdb snapshot folder need to be in pairs.

[0371] Understandably, Fig.16 The xxx.csv.txt file included in the blockdb snapshot folder is obtained by using a hash algorithm (such as the sha256 hash algorithm) to hash the xxx.csv file after the xxx.csv file is successfully stored on the disk, and the calculated hash data is stored in the obtained file. The corresponding relationship can be seen in Table 3. At the same time, in order to ensure data security, the xxx.csv file and the xxx.csv.txt file included in the blockdb snapshot folder need to be paired. The statedb snapshot folder is similar to the blockdb snapshot folder, and so on.

[0372] As mentioned above, the log structure merge tree has the problem of write amplification, and the simple copying also has the problem of write amplification; therefore, in the embodiment of the present application, by traversing all key-values ​​and storing the latest value of the same key in the xxx.csv file, the amount of data is reduced to achieve the effect of data slimming. The structure of the data in the xxx.csv file has only two columns, key and value. Each xxx.csv file stores a specified number (for example, 100,000) of key-value pairs, and the key-values ​​exceeding the number can be stored in a new xxx.csv file.

[0373] It is understandable that the naming rule of the xxx.csv file name can be x_yyyy.csv, where x represents the number of the xxx.csv file, and yyyy represents the first key in the xxx.csv file; for example, 1_stateDBSavepointKey.csv represents the first csv file, and the first key in the file is stateDBSavepointKey.

[0374] Exemplarily, multiple xxx.fdb.7z files may be generated in parallel, and / or multiple xxx.csv files may be generated in parallel.

[0375] See also Fig.17 , is a schematic diagram of multiple shard areas. Fig.17 As shown, the multiple shard regions include region1, region2...regionN. Accordingly, the key-value pairs included in the index file can be stored in the multiple shard regions region1, region2...regionN; therefore, in an optional embodiment, multiple xxx.csv files can be generated in parallel.

[0376] For example, blockchain node 1 may be interrupted during the snapshot process (such as network disconnection, downtime, etc.). Therefore, if the block height of the blockchain corresponding to blockchain node 1 does not increase at this time, the snapshot can be continued from the interruption position.

[0377] Among them, for the bfdb snapshot folder under snapshot_xxx_yyy, you can determine the interrupted xxx.fdb file through the file name of the generated xxx.fdb file, and then generate it from the interrupted xxx.fdb file. If xxx.fdb.7z and xxx.fdb.txt are not a pair, you can regenerate the corresponding file and use the regenerated corresponding file to overwrite the original incomplete corresponding file (see 1.5 for details). Optionally, you can also compare Fig.16 bfdb file list and Fig.15In the bfdb file list, determine the missing xxx.fdb.7z file.

[0378] Among them, for the kvdb snapshot folder under snapshot_xxx_yyy, you can determine the interrupted xxx.csv file through the file name of the generated xxx.csv file, and then generate it from the interrupted xxx.csv file. If xxx.csv and xxx.csv.txt are not a pair, you can regenerate the corresponding file and use the regenerated corresponding file to overwrite the original incomplete corresponding file (see 1.5 for details).

[0379] 1.5. Blockchain node 1 waits for all files to be generated and then verifies data integrity.

[0380] For example, after all files are generated, blockchain node 1 can detect whether the files in each folder are paired, for example, whether the xxx.fdb.7z file and the xxx.fdb.txt file are paired, and whether the xxx.csv file and the xxx.csv.txt file are paired. If it is found that there is no pair, the corresponding file can be regenerated and the original incomplete corresponding file can be overwritten with the regenerated corresponding file.

[0381] Among them, for regenerating xxx.fdb file, you need to know the startHeight and endHeight of this generation to determine the height range of the newly generated xxx.fdb file. The startHeight and endHeight can be determined by the file name. Optionally, for the files in the bfdb snapshot folder, you can also compare Fig.16 bfdb file list and Fig.15 Check whether the file list in bfdb is consistent.

[0382] Among them, for regenerating the xxx.csv file, it is necessary to know the startKey and endKey of the current generation to determine the key range of the xxx.csv file to be newly generated. The startKey and endKey can be determined by the file name.

[0383] See also Fig.18 , is a schematic diagram of the key range of the newly generated xxx.csv file. Fig.18 As shown, the key element interval corresponding to the newly generated 1_stateDBSavepointKey.csv file is [stateDBSavepointKey, theightKey].

[0384] 1.6. Blockchain node 1 generates connection information and generates a connection file based on the connection information.

[0385] For example, continuing with the previous example, blockchain node 1 also needs to store its own connection information in the peer.txt file so that blockchain nodes in need (such as blockchain node 2) can connect to itself and complete data verification.

[0386] 1.7. After blockchain node 1 generates all files, it can compress the connection file and the snapshot file, perform hash operation on the compressed file to obtain hash data corresponding to the compressed file, generate a snapshot verification file based on the hash data corresponding to the compressed file, and store the snapshot file and the snapshot verification file in association.

[0387] For example, following the above Fig.16 For example, compress the connection file and snapshot file into snapshot_chain1_3000.zip (or snapshot_chain1_3000.tar), perform hash operation on the snapshot_chain1_3000.zip file, and store the obtained hash data in the snapshot_chain1_3000.txt file; at the same time, store the snapshot_chain1_3000.zip file in the storage warehouse, and store the snapshot_chain1_3000.txt file in the on-chain storage repository.

[0388] For ease of understanding, the data import process (ie, S1304 to S1306) is described in detail below.

[0389] 2.1. Blockchain node 2 downloads the compressed file from the data warehouse, performs a hash operation on the compressed file, obtains the hash data corresponding to the compressed file, and decompresses the compressed file to obtain a connection file. It establishes communication with blockchain node 1 based on the connection information in the connection file, obtains a snapshot verification file from blockchain node 1 based on the communication link, and verifies the snapshot file based on the obtained hash data and the obtained snapshot verification file.

[0390] If the snapshot file passes the verification, the bfdb snapshot folder and the kvdb snapshot folder are imported. If the snapshot file fails the verification, further verification can be performed (see 2.2 for details).

[0391] For example, if the snapshot_chain1_3000.zip file is hashed to generate hash data corresponding to the snapshot_chain1_3000.zip file, if the hash data corresponding to the generated snapshot_chain1_3000.zip file matches the hash data in the snapshot_chain1_3000.txt file, it is determined that the verification of the snapshot_chain1_3000.zip file has passed. After that, importing the bfdb snapshot folder is to decompress xxx.fdb.7z in the bfdb file in the .zip decompressed file into the bfdb folder of blockchain node 2, and importing the kvdb snapshot folder is to read the data in sequence according to the sequence number of xxx.csv decompressed from .zip, and write it into the corresponding folder.

[0392] 2.2. The archive file is verified based on the archive verification file corresponding to the archive file, and the key-value file is verified based on the key-value verification file corresponding to the key-value file.

[0393] If the archive file passes the verification, the archive file is imported; if the archive file fails the verification, the archive file does not need to be imported.

[0394] For example, a hash operation is performed on the xxx.fdb.7z file to generate hash data corresponding to the xxx.fdb.7z file. If the hash data corresponding to the generated xxx.fdb.7z file matches the hash data in the xxx.fdb.txt file, it is determined that the verification of the xxx.fdb.7z file has passed, and the xxx.fdb.7z file is imported. If the hash data corresponding to the generated xxx.fdb.7z file does not match the hash data in the xxx.fdb.txt file, it is determined that the verification of the xxx.fdb.7z file has failed.

[0395] If the key-value file passes the verification, the key-value file is imported; if the key-value file fails the verification, the key-value file does not need to be imported.

[0396] For example, a hash operation is performed on the xxx.csv file to generate hash data corresponding to the xxx.csv file. If the hash data corresponding to the generated xxx.csv file matches the hash data in the xxx.csv.txt file, it is determined that the verification of the xxx.csv file has passed, and the xxx.csv file is imported. If the hash data corresponding to the generated xxx.csv file does not match the hash data in the xxx.csv.txt file, it is determined that the verification of the xxx.csv file has failed.

[0397] It should be noted that some of the introductions in the embodiments of the present application refer to the aforementioned embodiments and will not be repeated here.

[0398] The embodiments of the present application have at least the following beneficial effects:

[0399] (1) Generate a snapshot file. Then, you can use the snapshot file to perform import operations to repair data damage, improve data, etc.

[0400] (2) Search the index file for the latest value element corresponding to the same key element, and generate the key-value file corresponding to the index file based on the key-value pair matching the latest value element. The generated key-value file has a small data volume, avoiding the phenomenon of write amplification, making the data volume of the snapshot file small; at the same time, after compression, the data volume of the snapshot file is further reduced. Therefore, due to the small data volume of the snapshot file, storage space and transmission time are saved, and the efficiency of data snapshot and data import in the blockchain is improved.

[0401] (3) A verification file for the corresponding file will be generated, for example, a snapshot verification file corresponding to the snapshot file, a key-value verification file corresponding to the key-value file, and a verification file corresponding to the archive file. Thus, the verification file of the corresponding file can be used to implement verification, thereby improving the security of data snapshots and data import in the blockchain.

[0402] (4) It supports breakpoint continuation, further improving the efficiency of data snapshots in the blockchain, greatly improving the performance of the blockchain system, and is suitable for a wide range of scenarios.

[0403] Fig.19 FIG. 1 is a block diagram of a data processing device based on blockchain, which is shown in an embodiment of the present application. Fig.19 As shown, the blockchain-based data processing device includes:

[0404] The acquisition module 1901 is configured to acquire, if a data snapshot request is received, an archive file containing the chained blocks and an index file containing a key-value pair corresponding to the chained blocks, wherein the key-value pair includes a key element and a value element;

[0405] The first generating module 1902 is configured to search the index file for the latest value element corresponding to the same key element, and generate a key-value file corresponding to the index file based on the key-value pair matching the latest value element;

[0406] A second generating module 1903 is configured to generate a snapshot file based on the key-value file and the archive file, and generate a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file;

[0407] The storage module 1904 is configured to store the snapshot file and the snapshot verification file in association with each other.

[0408] In one embodiment of the present application, the key-value pairs contained in the index file are stored in the storage area in the form of a log-structured merge tree, and the log-structured merge tree includes multiple merge layers; the first generation module 1902 is specifically configured as follows:

[0409] The latest value element corresponding to the same key element is searched from the storage area in descending order of the number of merge layers; wherein the number of merge layers is inversely proportional to the writing time of the key-value pair.

[0410] In one embodiment of the present application, the second generation module 1902 is specifically configured as follows:

[0411] Compressing the key-value file, and performing a hash operation on the compressed key-value file to obtain hash data corresponding to the compressed key-value file;

[0412] Generate a key-value verification file for verifying the key-value file based on the hash data corresponding to the key-value file;

[0413] A snapshot file is generated based on the archive file, the compressed key-value file, and the key-value verification file.

[0414] In one embodiment of the present application, the compressed key-value files include multiple files, and the key-value verification files include multiple files; the second generation module 1902 is further specifically configured as follows:

[0415] For each compressed key-value file, the compressed key-value file and the key-value verification file corresponding to the compressed key-value file are associated through a preset file name definition strategy;

[0416] The archive file, the associated key-value file and the key-value verification file are combined to obtain a snapshot file.

[0417] In one embodiment of the present application, the blockchain-based data processing device further includes a third generation module configured to:

[0418] If it is detected that there is a target key-value file that is not associated with the key-value verification file in the snapshot file, regenerate the target key-value file based on the detected target key-value file, and regenerate the target key-value verification file corresponding to the newly generated target key-value file, and associate the newly generated target key-value file with the target key-value verification file, and replace the detected target key-value file with the associated target key-value file and target key-value verification file;

[0419] If it is detected that there is a target key value verification file in the snapshot file that is not associated with the key value file, the target key value file is regenerated based on the detected target key value verification file, and the target key value verification file corresponding to the newly generated target key value file is regenerated, and the newly generated target key value file is associated with the target key value verification file, and the detected target key value verification file is replaced with the associated target key value file and target key value verification file.

[0420] In one embodiment of the present application, the third generation module is specifically configured as follows:

[0421] Based on the file name corresponding to the detected target key-value file and the file names corresponding to the key-value files adjacent to the detected target key-value file, determine the key element interval corresponding to the target key-value file to be newly generated, and regenerate the target key-value file based on the key element interval;

[0422] Based on the file name corresponding to the detected target key-value verification file, determine the file name corresponding to the target key-value file to be newly generated; based on the file name corresponding to the target key-value file to be newly generated and the file names corresponding to key value files adjacent to the target key-value file to be newly generated, determine the key element interval corresponding to the target key-value file to be newly generated; and regenerate the target key-value file based on the key element interval.

[0423] In one embodiment of the present application, the second generation module 1902 is specifically configured as follows:

[0424] Compressing the archive file, and performing a hash operation on the compressed archive file to obtain hash data corresponding to the compressed archive file;

[0425] Generate an archive verification file for verifying the archive file based on the hash data corresponding to the archive file;

[0426] A snapshot file is generated based on the key-value file, the compressed archive file, and the archive verification file.

[0427] In one embodiment of the present application, the key-value pairs contained in the index file are stored in multiple shard areas; the first generation module 1902 is specifically configured as follows:

[0428] The latest value element corresponding to the same key element is searched in parallel from each shard area, and a key-value file corresponding to the index file is generated based on the key-value pair matching the latest value element.

[0429] In one embodiment of the present application, the index file includes a plurality of index files, and the blockchain-based data processing device further includes a fourth generation module configured to:

[0430] If an interruption is detected in the process of generating the key-value file corresponding to the index file, the starting value corresponding to the key element to be searched is determined based on the file name corresponding to the generated key-value file;

[0431] The latest value element corresponding to the starting value is searched starting from the starting value, and a key-value file corresponding to the index file is generated based on the key-value pair matching the latest value element corresponding to the starting value, until key-value files corresponding to the multiple index files are generated.

[0432] In one embodiment of the present application, the second generation module 1903 is specifically configured as follows:

[0433] Compressing the snapshot file, and performing a hash operation on the compressed snapshot file to obtain hash data corresponding to the compressed snapshot file;

[0434] A snapshot verification file for verifying the snapshot file is generated based on the hash data corresponding to the snapshot file.

[0435] In one embodiment of the present application, the storage module 1904 is specifically configured as follows:

[0436] The compressed snapshot file and the snapshot verification file are stored in association with each other.

[0437] Fig. 20 FIG. 1 is a block diagram of a data processing device based on blockchain, which is shown in an embodiment of the present application. Fig. 20 As shown, the blockchain-based data processing device includes:

[0438] The first acquisition module 2001 is configured to acquire the associated stored snapshot file and snapshot verification file if a data import request is received; wherein the snapshot file includes a key-value file and an archive file, and the key-value file is generated based on a key-value pair matching the latest value element corresponding to the same key element;

[0439] A second acquisition module 2002 is configured to acquire the key-value file and the archive file from the snapshot file if the snapshot file is verified based on the snapshot verification file;

[0440] The import module 2003 is configured to import the key-value file and the archive file into the local storage area.

[0441] In one embodiment of the present application, the key-value files include multiple ones, and the blockchain-based data processing device also includes a first verification module configured to:

[0442] If the snapshot file fails to be verified based on the snapshot verification file, obtaining a key-value verification file corresponding to each key-value file from the snapshot file;

[0443] For each key-value file, verify the key-value file based on the key-value verification file corresponding to the key-value file;

[0444] Import the verified key-value files into the local storage area.

[0445] In one embodiment of the present application, the archived files include multiple files, and the blockchain-based data processing device also includes a second verification module configured to:

[0446] If the snapshot file fails to be verified based on the snapshot verification file, obtaining an archive verification file corresponding to each archive file from the snapshot file;

[0447] For each archive file, verify the archive file based on the archive verification file corresponding to the archive file;

[0448] Import the verified archived files into the local storage area.

[0449] In one embodiment of the present application, the first acquisition module 2001 is specifically configured as follows:

[0450] Acquire a snapshot file from the snapshot party, and decompress the snapshot file to obtain connection information for establishing communication with the snapshot party;

[0451] A snapshot verification file corresponding to the snapshot file is obtained from the snapshot party based on the connection information.

[0452] It should be noted that the apparatus provided in the aforementioned embodiment and the method provided in the aforementioned embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment.

[0453] An embodiment of the present application also provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the blockchain-based data processing method as described above.

[0454] Fig.21 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application.

[0455] It should be noted that Fig.21 The computer system 2100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0456] like Fig.21As shown, the computer system 2100 includes a central processing unit (CPU) 2101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 2102 or the program loaded from the storage part 2108 to the random access memory (RAM) 2103, such as executing the method in the above embodiment. In the RAM 2103, various programs and data required for system operation are also stored. The CPU 2101, ROM 2102 and RAM 2103 are connected to each other through a bus 2104. An input / output (I / O) interface 2105 is also connected to the bus 2104.

[0457] The following components are connected to the I / O interface 2105: an input section 2106 including a keyboard, a mouse, etc.; an output section 2107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 2108 including a hard disk, etc.; and a communication section 2109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 2109 performs communication processing via a network such as the Internet. A drive 2110 is also connected to the I / O interface 2105 as needed. A removable medium 2111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 2110 as needed so that a computer program read therefrom is installed into the storage section 2108 as needed.

[0458] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 2109, and / or installed from a removable medium 2111. When the computer program is executed by a central processing unit (CPU) 2101, various functions defined in the system of the present application are executed.

[0459] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable media may be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0460] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0461] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0462] Another aspect of the present application further provides a computer-readable medium having a computer program stored thereon, which implements the blockchain-based data processing method as described above when the computer program is executed by a processor. The computer-readable medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0463] Another aspect of the present application further provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable medium. The processor of the computer device reads the computer instruction from the computer-readable medium, and the processor executes the computer instruction, so that the computer device executes the data processing method based on blockchain provided in each of the above embodiments.

[0464] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A data processing method based on blockchain, characterized in that: include: If a data snapshot request is received, an archive file containing the chained blocks and an index file containing key-value pairs corresponding to the chained blocks are obtained, wherein the key-value pairs include a key element and a value element; Searching for the latest value element corresponding to the same key element in the index file, and generating a key-value file corresponding to the index file based on the key-value pair matching the latest value element; Generate a snapshot file based on the key-value file and the archive file, and generate a snapshot verification file for verifying the snapshot file based on hash data of the snapshot file; The snapshot file and the snapshot verification file are stored in association with each other.

2. The method according to claim 1, characterized in that The key-value pairs contained in the index file are stored in the storage area in the form of a log-structured merge tree, and the log-structured merge tree includes multiple merge layers; searching for the latest value element corresponding to the same key element from the index file includes: The latest value element corresponding to the same key element is searched from the storage area in descending order of the number of merge layers; wherein the number of merge layers is inversely proportional to the writing time of the key-value pair.

3. The method according to claim 1, characterized in that The generating a snapshot file based on the key-value file and the archive file includes: Compressing the key-value file, and performing a hash operation on the compressed key-value file to obtain hash data corresponding to the compressed key-value file; Generate a key-value verification file for verifying the key-value file based on the hash data corresponding to the key-value file; A snapshot file is generated based on the archive file, the compressed key-value file, and the key-value verification file.

4. The method according to claim 3, characterized in that The compressed key-value files include a plurality of files, and the key-value verification files include a plurality of files; and generating a snapshot file based on the archive file, the compressed key-value files, and the key-value verification files includes: For each compressed key-value file, the compressed key-value file and the key-value verification file corresponding to the compressed key-value file are associated through a preset file name definition strategy; The archive file, the associated key-value file and the key-value verification file are combined to obtain a snapshot file.

5. The method according to claim 4, characterized in that After combining the archive file, the associated key-value file and the key-value verification file to obtain a snapshot file, the method further includes: If it is detected that there is a target key-value file that is not associated with the key-value verification file in the snapshot file, regenerate the target key-value file based on the detected target key-value file, and regenerate the target key-value verification file corresponding to the newly generated target key-value file, and associate the newly generated target key-value file with the target key-value verification file, and replace the detected target key-value file with the associated target key-value file and target key-value verification file; If it is detected that there is a target key value verification file in the snapshot file that is not associated with the key value file, the target key value file is regenerated based on the detected target key value verification file, and the target key value verification file corresponding to the newly generated target key value file is regenerated, and the newly generated target key value file is associated with the target key value verification file, and the detected target key value verification file is replaced with the associated target key value file and target key value verification file.

6. The method according to claim 5, characterized in that The regenerating the target key-value file based on the detected target key-value file comprises: Based on the file name corresponding to the detected target key-value file and the file names corresponding to the key-value files adjacent to the detected target key-value file, determine the key element interval corresponding to the target key-value file to be newly generated, and regenerate the target key-value file based on the key element interval; The regenerating the target key-value file based on the detected target key-value verification file comprises: Based on the file name corresponding to the detected target key-value verification file, determine the file name corresponding to the target key-value file to be newly generated; based on the file name corresponding to the target key-value file to be newly generated and the file names corresponding to key value files adjacent to the target key-value file to be newly generated, determine the key element interval corresponding to the target key-value file to be newly generated; and regenerate the target key-value file based on the key element interval.

7. The method according to claim 1, characterized in that The generating a snapshot file based on the key-value file and the archive file includes: Compressing the archive file, and performing a hash operation on the compressed archive file to obtain hash data corresponding to the compressed archive file; Generate an archive verification file for verifying the archive file based on the hash data corresponding to the archive file; A snapshot file is generated based on the key-value file, the compressed archive file, and the archive verification file.

8. The method according to any one of claims 1 to 7, characterized in that The key-value pairs contained in the index file are stored in multiple shard areas; searching the index file for the latest value element corresponding to the same key element, and generating a key-value file corresponding to the index file based on the key-value pair matching the latest value element, includes: The latest value element corresponding to the same key element is searched in parallel from each shard area, and a key-value file corresponding to the index file is generated based on the key-value pair matching the latest value element.

9. The method according to any one of claims 1 to 7, characterized in that The index file includes a plurality of index files, and the method further includes: If an interruption is detected in the process of generating the key-value file corresponding to the index file, the starting value corresponding to the key element to be searched is determined based on the file name corresponding to the generated key-value file; The latest value element corresponding to the starting value is searched starting from the starting value, and a key-value file corresponding to the index file is generated based on the key-value pair matching the latest value element corresponding to the starting value, until key-value files corresponding to the multiple index files are generated.

10. The method according to any one of claims 1 to 7, characterized in that The generating a snapshot verification file for verifying the snapshot file based on the hash data of the snapshot file includes: Compressing the snapshot file, and performing a hash operation on the compressed snapshot file to obtain hash data corresponding to the compressed snapshot file; Generate a snapshot verification file for verifying the snapshot file based on the hash data corresponding to the snapshot file; The storing the snapshot file and the snapshot verification file in association with each other includes: The compressed snapshot file and the snapshot verification file are stored in association with each other.

11. A data processing method based on blockchain, characterized in that: include: If a data import request is received, the associated stored snapshot file and snapshot verification file are obtained; wherein the snapshot file includes a key-value file and an archive file, and the key-value file is generated based on a key-value pair matching the latest value element corresponding to the same key element; If the snapshot file is verified based on the snapshot verification file, the key-value file and the archive file are obtained from the snapshot file; The key-value file and the archive file are imported into a local storage area.

12. The method according to claim 11, characterized in that The key-value files include a plurality of key-value files, and the method further includes: If the snapshot file fails to be verified based on the snapshot verification file, obtaining a key-value verification file corresponding to each key-value file from the snapshot file; For each key-value file, verify the key-value file based on the key-value verification file corresponding to the key-value file; Import the verified key-value files into the local storage area.

13. The method according to claim 11, characterized in that The archive file includes a plurality of files, and the method further includes: If the snapshot file fails to be verified based on the snapshot verification file, obtaining an archive verification file corresponding to each archive file from the snapshot file; For each archive file, verify the archive file based on the archive verification file corresponding to the archive file; Import the verified archived files into the local storage area.

14. The method according to any one of claims 11 to 13, characterized in that The obtaining of the associated stored snapshot file and the snapshot verification file includes: Acquire a snapshot file from the snapshot party, and decompress the snapshot file to obtain connection information for establishing communication with the snapshot party; A snapshot verification file corresponding to the snapshot file is obtained from the snapshot party based on the connection information.

15. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the electronic device, enables the electronic device to implement the blockchain-based data processing method as described in any one of claims 1 to 14.

16. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the blockchain-based data processing method according to any one of claims 1 to 14 is implemented.

17. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the blockchain-based data processing method as described in any one of claims 1 to 14 is implemented.

Images