A block data processing method, device and electronic equipment

Abstract

The application discloses a kind of block data processing method, device and electronic equipment, method includes: each block data in the first original file of storage block data is added first check data section and block length field, obtains block file;And each block index information in the second original file is added second check data section and index length field, obtains index file;In response to the file processing request of block file, according to first check data section, block length field and block data, each block file is processed corresponding, obtains file processing result;And according to file processing result, in combination with second check data section, index length field and index data, each index file is updated.The application can improve the processing efficiency of block data and reduce the security risk of data processing process, and can be widely applied in cloud computing, Internet of Vehicles and other technical fields related to Internet technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a block data processing method, apparatus, and electronic device. Background Technology

[0002] As blockchain technology becomes increasingly widespread, the scale of individual blockchains is also expanding rapidly, and new data is generated at a fast pace. Block data consists of raw data uploaded by nodes in each blockchain. During blockchain applications, block data only increases and never decreases. Moreover, once block data is written, it must not be modified to ensure data consistency among blockchain nodes. This results in significant data storage and management pressures.

[0003] Related technologies typically store all block data in the same file, which means that every time you write or delete block data, you have to operate on the blocks in this file. The processing dimension stays at the block data stage, resulting in inefficient data storage operations. Furthermore, the corruption of a single block of data can easily lead to the corruption of the entire storage file, posing a threat to data security. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a block data processing method, apparatus, and electronic device to improve the efficiency of data storage operations and enhance the security of data storage in the blockchain.

[0005] On one hand, embodiments of the present invention provide a block data processing method, including the following steps:

[0006] Add a first checksum data segment and a block length field to each block of data in the first original file storing the block data to obtain the block file;

[0007] A second check data segment and an index length field are added to each block index information in the second original file that stores the block index information to obtain the index file;

[0008] In response to file processing requests for block files, each block file is processed according to the first checksum data segment, block length field, and block data in the block file to obtain the file processing result; among which, file processing requests include file read requests, file write requests, file merge requests, and file split requests;

[0009] Based on the file processing results, and in conjunction with the second checksum data segment, index length field, and index data in the index files, each index file is updated.

[0010] On the other hand, embodiments of the present invention also provide a block data processing apparatus, including:

[0011] The first module is used to add a first check data segment and a block length field to each block data in the first original file storing block data to obtain a block file;

[0012] The second module is used to add a second check data segment and an index length field to each block index information in the second original file storing the block index information, so as to obtain an index file;

[0013] The third module is used to respond to file processing requests for block files. Based on the first checksum data segment, block length field and block data in the block file, it performs corresponding processing on each block file to obtain the file processing result. The file processing requests include file read requests, file write requests, file merge requests and file split requests.

[0014] The fourth module is used to update each index file based on the file processing results, combined with the second checksum data segment, index length field, and index data in the index file.

[0015] Optionally, the first verification data segment is used to verify the integrity of the block file; the block length field is used to characterize the field length of the block data in the block file; the second verification data segment is used to verify the integrity of the index file; the index length field is used to characterize the field length of the index information in the index file; the first module is specifically used for:

[0016] Get the storage size of the block data;

[0017] Calculate the block length of the block data based on the storage size, and add a block length field to the block data in the first original file.

[0018] Optionally, when the file processing request is a file read request, the third module is specifically used for:

[0019] In response to a file read request for a block file, the file header of the block file is obtained, and the first checksum data segment and the block length field are read sequentially from the file header;

[0020] Based on the block length field, read data of the corresponding length from the block file to obtain the block data;

[0021] The verification information of the block data is calculated and compared with the first verification data segment to determine the integrity of the block data, until all block data in the block file is read.

[0022] Optionally, the device further includes:

[0023] The fifth module is used to terminate the reading of the block file and discard all content after the erroneous data when the block data being read is erroneous.

[0024] Optionally, when the file processing request is a file write request, the third module is specifically used for:

[0025] In response to a file write request to the block file, the first checksum data segment, the block length field, and the block data are written to the block file in sequence.

[0026] The index information corresponding to each block of data is written to the index file. After all the block data has been written, an end marker is added to the index file. The end marker is used to indicate the last block of data written to the block file.

[0027] Optionally, the device further includes:

[0028] The sixth module is used to terminate the writing process to the block file when a write error occurs while writing block data to the block file, and to read the end marker when restarting the writing process, and rewrite the block data starting from the last block data.

[0029] Optionally, when the file processing request is a file merge request, the third module is specifically used for:

[0030] In response to a file merging request for block files, obtain several time-adjacent block files;

[0031] The first block file among several adjacent block files is used as the merge file, and the other block files are merged into the merge file in sequence; wherein, each block data in the merge file includes a first check data segment and a block length field;

[0032] Based on the information of the new block data in the merged file, the index file of the first block file is updated to obtain the index file corresponding to the merged file;

[0033] Store the filenames of the individual block files to be merged into the target file;

[0034] Based on the target file's filename, delete the corresponding block files and index files from the temporary folder; the temporary folder is used to store files during the processing.

[0035] Optionally, the third module is further used for:

[0036] Based on the filenames of each block file, the block height of each block file is parsed, and based on the block height of each block file, it is determined that each block file is a temporally adjacent block file.

[0037] The first block file among the sequentially adjacent block files is used as the merge file, and the other block files are merged into the merge file in sequence.

[0038] Optionally, when the file processing request is a file merge request, the third module is further used for:

[0039] When a merge error occurs while merging block files, check the file types present in the temporary folder:

[0040] When the temporary folder contains a merge file, an index file, and a target file, the merge file, index file, and target file are repaired before the file merging of the block file is completed.

[0041] If the temporary folder contains only merged files, delete the merged files to complete the file merging of the block files.

[0042] When there is no merge file in the temporary folder, but there is an index file and a target file, the data in the index file is written to the original folder, and according to the file name of the target file, the corresponding block files and index files are deleted from the temporary folder to complete the file merging of the block files;

[0043] When the temporary folder contains only the target file, delete the corresponding block files and index files from the temporary folder based on the target file's filename, thus completing the file merging of the block files.

[0044] Optionally, when the file processing request is a file splitting request, the third module includes:

[0045] The sixth unit is used to determine the block file's segmentation height and the number of segments in response to a file segmentation request for the block file;

[0046] The seventh unit is used to divide the block file into a corresponding number of sub-files based on the segmentation height and the number of segmented files; wherein each block data in the sub-file includes a first check data segment and a block length field;

[0047] Unit 8 is used to configure the index file corresponding to each sub-file obtained from the split.

[0048] Optionally, the sixth unit is specifically used for at least one of the following:

[0049] Identify the overall block height of the block file and determine the height of each segment from the overall block height based on the number of segmented files parsed in the file segmentation request;

[0050] or,

[0051] Calculate the data size of the block file, and determine the number of files to be split and the block file split height based on the size of the file to be split parsed in the file split request.

[0052] Optionally, the device further includes:

[0053] The seventh module is used to respond to file splitting requests for block files and record the file processing status of the block file as the file splitting status;

[0054] The eighth module is used to change the file processing status to file writing status when the file splitting process of the block file is completed.

[0055] Optionally, when the file processing request is a file splitting request, the third module is further used for:

[0056] If a splitting error occurs while splitting a block file, stop the file splitting operation and delete the generated sub-files and their corresponding index files.

[0057] When a blockchain network failure occurs during the file splitting of a block file, data repair is performed based on the file processing status of the block file until the file splitting process is completed.

[0058] Optionally, when the file processing status of the block file is in the file splitting state, the third module is further used for:

[0059] When each sub-file obtained from the block file segmentation exists in pairs with its corresponding index file, the sub-files are repaired according to the index file, and the block file segmentation process is restarted.

[0060] When a block file is split into pairs of sub-files and index files, and a single sub-file is obtained, delete the single sub-file and restart the file splitting process for the block file.

[0061] If the block file is split into only one sub-file, restart the block file splitting process directly.

[0062] Optionally, when the block file's file processing status is in file write status, the third module is further used for:

[0063] When each sub-file obtained from the block file segmentation exists in pairs with its corresponding index file, the sub-files are transferred to the block management database, and the file segmentation process for the block file is restarted; the block management database is used to store the processed files.

[0064] When a block file is split into pairs of sub-files and an index file, and a separate index file is obtained, the index file is written to the index database, and the file splitting process for the block file is restarted; the index database is used to store the index information of the block data.

[0065] On the other hand, embodiments of the present invention also provide an electronic device, including a processor and a memory;

[0066] Memory is used to store programs;

[0067] The processor executes the program to implement the block data processing methods described above.

[0068] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a program, which is executed by a processor to implement the block data processing method described above.

[0069] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the aforementioned block data processing method.

[0070] In embodiments of the present invention, a first checksum and a block length field are added to each block data in a first original file storing block data to obtain a block file. Then, a second checksum and an index length field are added to each block index information in a second original file storing block index information to obtain an index file. The present invention restructures existing block data storage files by adding corresponding checksum and length fields, enabling subsequent file processing based on requests to directly process each file at the file level, without needing to process at the block data level. Then, based on the structurally optimized block file and index file, the present invention responds to... The block file processing request, based on the first checksum segment, block length field, and block data in the block file, processes each block file accordingly to obtain the file processing result; and based on the file processing result, combined with the second checksum segment, index length field, and index data in the index file, updates each index file. This invention targets the file dimension for processing, without splitting multiple blocks of data stored within a file for separate processing, which can improve the processing efficiency of block data. At the same time, because it increases the dimension of the data processing object, this invention can also reduce the security risks of the data processing process, and the processing of a single block of data will not cause the entire block file to become invalid. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1This is a schematic diagram of the structure of a blockchain node system provided in an embodiment of the present invention;

[0073] Figure 2 This is a flowchart illustrating the steps of a block data processing method provided in an embodiment of the present invention.

[0074] Figure 3 This is a schematic diagram of the directory structure for storing block file data provided in an embodiment of the present invention;

[0075] Figure 4 This is a schematic diagram of the composition structure of the first original file provided in an embodiment of the present invention;

[0076] Figure 5 This is a schematic diagram of the composition structure of a block file provided in an embodiment of the present invention;

[0077] Figure 6 This is a schematic diagram of the composition structure of the index file provided in an embodiment of the present invention;

[0078] Figure 7 This is a schematic diagram of one embodiment of merging xxx.fdb files provided in this invention.

[0079] Figure 8 A schematic diagram of an embodiment of the present invention, which involves merging multiple xxx.fdb files into a merge folder.

[0080] Figure 9 A flowchart illustrating an embodiment of a block file merging error provided by this invention;

[0081] Figure 10 This is a schematic diagram of the process for handling errors in xxx.fdb merging provided in an embodiment of the present invention;

[0082] Figure 11 A schematic diagram illustrating one embodiment of file segmentation for xxx.fdb provided in this invention;

[0083] Figure 12 A schematic diagram illustrating an embodiment of the present invention where the xxx.fdb file is split into a split folder;

[0084] Figure 13 A flowchart illustrating an embodiment of a block file segmentation error provided by this invention;

[0085] Figure 14 This is a schematic diagram of the error handling process during the segmentation of xxx.fdb provided in an embodiment of the present invention;

[0086] Figure 15This is a modular schematic diagram of a block data processing device provided in an embodiment of the present invention;

[0087] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0088] Figure 17 This is a schematic diagram of the computer system structure of an electronic device implementing the embodiments of this application, provided for the purposes of this invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0090] It should be understood that in the description of the embodiments of the present invention, "multiple" (or "amounts") means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If "first," "second," etc., are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0091] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0092] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms that may be involved in the embodiments of the present invention are explained, and the nouns and terms involved in the embodiments of the present invention are subject to the following interpretations:

[0093] Blockchain: A distributed ledger technology in the field of information technology, generally composed of consensus, transaction blocks and state data storage, and cryptographic identity security. Because the ledger is distributed and the blocks are agreed upon, it has characteristics such as immutability, traceability, and joint maintenance. A blockchain is a chain composed of several blocks. Each block stores certain information, and they are linked together in the order they were generated. This chain is stored on all servers; as long as one server in the entire system is operational, the entire blockchain is secure. These servers are called nodes in the blockchain system, and they provide storage space and computing power support for the entire blockchain system. To modify information in the blockchain, the consent of more than half of the nodes must be obtained, and the information in all nodes must be modified. These nodes are usually controlled by different entities, making it extremely difficult to tamper with information in the blockchain. Compared to traditional networks, blockchain has two core characteristics: first, the data is difficult to tamper with; second, it is decentralized. Based on these two characteristics, the information recorded by the blockchain is more authentic and reliable, helping to solve the problem of mutual distrust.

[0094] Smart contracts: A type of program that runs on a blockchain. Unlike ordinary programs, smart contracts must ensure that the result is exactly the same on every node of the blockchain network, so that any node can verify whether the result of the smart contract execution in the block is correct.

[0095] Certificates: A digital certificate (or simply certificate) is an electronic document that uniquely identifies a person and a resource on the Internet. Certificates enable secure and confidential communication between two entities. There are many types of certificates, such as personal certificates (used by individuals) and server certificates (used to establish secure sessions between servers and clients using Secure Sockets Layer (SSL) technology).

[0096] Signature verification: A digital signature (also known as a public-key digital signature) is a string of numbers that only the sender of the information can generate and that cannot be forged by others. This string of numbers also serves as valid proof of the authenticity of the information sent by the sender. Signature verification is the process by which the recipient confirms that the encrypted file they received was indeed sent by the sender. It requires checking whether it has been tampered with (unlike decryption). Signature verification ultimately involves comparing the message digest.

[0097] Transaction Pool: The transaction pool is a pool where blockchain transactions are temporarily stored. Each transaction awaiting block generation first arrives at the transaction pool for temporary storage and is ordered according to a certain priority. When a new block is generated, transactions are retrieved from the transaction pool in order. The blockchain then executes and verifies the transactions, and finally, after all nodes have verified them, the transactions in the block are deleted from the transaction pool.

[0098] Transaction: A way to submit data to the blockchain.

[0099] Block: In most blockchain implementations, the unit of data storage on the chain is the block number, which is also the block height.

[0100] ChainMaker: Chang'an Chain is a technology system with outstanding features of independent controllability, flexible assembly, integrated hardware and software, and open source. It is designed to meet the needs of next-generation blockchain technology, such as large-scale node networking, high transaction processing performance, and strong data security and privacy. It integrates blockchain-specific acceleration chip hardware and an assembleable underlying software platform to provide new solutions for building high-performance, highly reliable, and highly secure digital infrastructure.

[0101] Uploading to the blockchain: Data is stored in the database of the blockchain, which is called uploading to the blockchain.

[0102] Node: A service in a blockchain. A blockchain is generally composed of multiple nodes, and each node starts a blockchain software service.

[0103] KVDB: This is the term used in this invention to refer to a non-relational database. Common KVDBs include LevelDB and BadgerDB. LevelDB, in particular, can support data volumes up to billion. It is a single-process service with extremely high performance; on a quad-core Q6600 CPU machine, it can write over 400,000 data entries per second, and its random read performance exceeds 100,000 per second.

[0104] CRC (Cyclic Redundancy Check) is a fast algorithm that generates a short, fixed-length check code based on data such as network packets or computer files. It is primarily used to detect or verify errors that may occur during data transmission or storage. CRC utilizes the principles of division and remainders to achieve error detection, and it has advantages such as clear principles and simple implementation.

[0105] Block file database (BFDB): A type of block management database used in ChainMaker, suitable for storing and managing block data in a blockchain that is written once and not changed subsequently. Multiple databases are typically built within a BFDB, each storing its own data (e.g., block transaction data, block index data, block state data, block history data, etc.). The files stored in different databases are independent and do not change.

[0106] See Figure 1 , Figure 1 This is a schematic diagram of a blockchain node system provided in an embodiment of the present invention. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. It is mainly used to organize data in chronological order and encrypt it into a ledger, making it tamper-proof and forgery-proof. It also allows for data verification, storage, and updating. Essentially, a blockchain is a decentralized database where each node stores an identical blockchain record. The blockchain network includes consensus nodes, which are responsible for achieving consensus across the entire blockchain network.

[0107] As can be understood, a block is a data packet that carries transaction data (i.e., transaction business) on a blockchain network. It is a data structure that is marked with a timestamp and the hash value of the previous block. The block is verified and the transactions in the block are confirmed by the network's consensus mechanism.

[0108] It is understandable that a blockchain system can include smart contracts. A smart contract in a blockchain system can be understood as code that can be understood and executed by all nodes in the blockchain (including consensus nodes), capable of executing arbitrary logic and obtaining results. It should be understood that a blockchain can include one or more smart contracts, which can be distinguished by an identity document (ID) or name. Transaction requests can carry the identity document or name of the smart contract to specify the smart contract that the blockchain needs to run.

[0109] like Figure 1 The blockchain node system 100 shown can correspond to a blockchain network, which may include, but is not limited to, the blockchain network corresponding to a consortium blockchain. The blockchain node system 100 refers to a system used for data sharing between blockchain nodes. This blockchain node system can include multiple nodes, specifically nodes 10a, 10b, 10c, ..., 10n, which can be collectively referred to as blockchain nodes.

[0110] Each node receives input information during normal operation and maintains the shared data within the blockchain node system based on this information. To ensure interoperability within the blockchain node system, information connections exist between each node, allowing for information transmission. For example, when any node in the blockchain node system receives input information, other nodes in the system retrieve this input information according to the consensus algorithm and store it as part of the shared data, ensuring data consistency across all nodes in the blockchain node system.

[0111] It is understood that the above-mentioned information connection is not limited to the connection method. It can be connected directly or indirectly through wired communication, or directly or indirectly through wireless communication, or through other connection methods. This invention does not impose any restrictions on these methods.

[0112] Understandably, a blockchain network can establish information connections between nodes based on node identifiers. Each node in a blockchain node system has a corresponding node identifier, and each node can store the node identifiers of other nodes in the blockchain node system. This allows the generated blocks to be broadcast to other nodes in the blockchain node system based on their node identifiers. Each node can maintain a node identifier list as shown in the table below, storing the node name and node identifier in this list.

[0113] Table 1

[0114] Node Name Node identifier Node 1 117.X1.Y1.Z1 Node 2 117.X2.Y2.Z2 … … Node N 119.Xn.Yn.Zn

[0115] The node identifier can be an IP (Internet Protocol) address or any other information that can be used to identify a node in the blockchain network. Table 1 only uses IP addresses as an example. For example, node 1 (e.g., node 1 can be...) Figure 1 Node 10a) shown can send a request to node 2 (e.g., node 2 can be a node with node identifier 117.116.189.145) via node identifier 117.116.189.145. Figure 1 Node 10b) sends information (e.g., blocks), and Node 2 can determine that the information was sent by Node 1 by using node identifier 117.114.151.174.

[0116] It is understood that the block data processing method provided in this embodiment of the invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in this embodiment of the invention is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet computer, laptop computer, or desktop computer, but it is not limited to these.

[0117] It should be further noted that the terminals involved in the embodiments of the present invention include, but are not limited to, smartphones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0118] It should be understood that in a blockchain, before a block (i.e., a proposed block, a block to be added to the chain) is added, it must pass consensus among the consensus nodes in the blockchain network. Only after consensus is reached can the block be added to the blockchain. It is understandable that when blockchain is used in scenarios involving government or commercial institutions, not all participating nodes in the blockchain (i.e., the blockchain nodes in the aforementioned blockchain node system 100) have sufficient resources and the necessity to become consensus nodes. For example, in... Figure 1 In the blockchain node system 100 shown, nodes 10a, 10b, and 10c can be used as consensus nodes in the blockchain node system 100. Node 10a can be used as a packaging node to generate blocks, and nodes 10b and 10c can be used as consensus nodes to participate in the consensus.

[0119] Understandably, the packaging node (which is a consensus node) of the blockchain can send the newly generated block (i.e., the proposed block) to other consensus nodes in the blockchain node system 100 based on the node identifiers of other consensus nodes in the blockchain node system 100. The consensus nodes then reach a consensus on the newly generated block and add it to their stored blockchain after successful consensus.

[0120] It should be understood that the first node and the second node in the embodiments of the present invention can be consensus nodes in a blockchain network. The first node can be a packaging node in the blockchain network. Therefore, in the embodiments of the present invention, the blockchain node that packages transaction data can be collectively referred to as the first node (i.e., the packaging node, for example, ...). Figure 1 As shown in node 10a), the blockchain nodes that reach consensus on the blocks obtained from the packaging process can also be collectively referred to as second nodes (i.e., consensus nodes, for example, Figure 1 (Nodes 10b and 10c are shown).

[0121] For better understanding, please refer to [link / reference]. Figure 2 , Figure 2 This is a flowchart illustrating a block data processing method provided by an embodiment of the present invention. The method can be implemented by a first node (e.g., the one described above). Figure 1 The blockchain node 10a shown can be executed by a second node (e.g., the one mentioned above). Figure 1The blockchain node 10b shown can be executed, and it can also be executed jointly by the first node and the second node.

[0122] The block data processing method may include at least the following steps S201 to S204:

[0123] Step S201: Add a first check data segment and a block length field to each block data in the first original file storing the block data to obtain the block file;

[0124] It should be understood that the method of this embodiment of the invention optimizes the structure of a first original file storing block data to enable secure and efficient block data processing based on the structural characteristics of the optimized block file. It should be noted that related structural optimization operations can be performed simultaneously or sequentially on several first original files under the main directory of the block data. Specifically, the structural optimization of the first original file includes adding a first checksum segment and a block length field to each block data in the file, thereby obtaining a block file. Both the first original file and the block file include several blocks of data, and each block data in the block file obtained after structural optimization carries the first checksum segment and the block length field.

[0125] It is understandable that the first verification data segment can be used for data verification in subsequent processing of related block data. Data verification is a validation operation performed to ensure data integrity. Typically, a specified algorithm is used to calculate a verification value for the original data. The receiver / processor calculates the verification value using the same algorithm. If the two calculated verification values ​​are the same, it indicates that the data is complete. In this embodiment of the invention, the first verification data segment corresponding to each block data is calculated separately using a specified algorithm. Since the block file in this embodiment of the invention includes the structural composition characteristics of several block data, the block length field can assist in accurately locating the field position of the processed data in subsequent processing of related block data.

[0126] For example, in an embodiment of the present invention, the first check data segment of each block data in the first original file can be calculated by using the CRC algorithm in ChainMaker, and the block length field can be determined based on the storage amount or byte size of each block data. Then, the obtained first check data segment and block length field are added to the corresponding block data. After all the block data in the first original file has the corresponding first check data segment and block length field added, a block file is obtained.

[0127] Step S202: Add a second check data segment and an index length field to each block index information in the second original file storing the block index information to obtain the index file;

[0128] It should be understood that the method of this embodiment of the invention optimizes the structure of the second original file storing block index information to facilitate secure and efficient block data processing based on the structural characteristics of the optimized index file. It should be noted that related structural optimization operations can be performed simultaneously or sequentially on several second original files under the main directory of the block index information (block index data). Specifically, the structural optimization of the second original file includes adding a second check data segment and an index length field to each block index information in the file, thereby obtaining an index file. Both the second original file and the index file include several block index information segments, while the index file obtained after structural optimization carries the second check data segment and the index length field for each block index information segment.

[0129] It is understandable that the second verification data segment can be used for data verification in subsequent processing of related block data based on block index information. Data verification is a validation operation to ensure data integrity. Typically, a verification value is calculated on the original data using a specified algorithm. The receiver / processor calculates the verification value using the same algorithm. If the two calculated verification values ​​are the same, it indicates that the data is complete. In this embodiment of the invention, the second verification data segment corresponding to each block index information is calculated using a specified algorithm. Because the block file in this embodiment of the invention includes the structural characteristics of several block index information, the index length field can assist subsequent processing of related block data based on block index information in accurately locating the field position of the referenced block index information.

[0130] For example, in embodiments of the present invention, the second verification data segment of each block index information in the second original file can be calculated using the CRC algorithm in a blockchain (e.g., ChainMaker), and the index length field can be determined based on the storage amount or byte size of each block index information. Then, the obtained second verification data segment and index length field are added to the corresponding block index information. After all the block index information in the second original file has the corresponding second verification data segment and index length field added, an index file is obtained.

[0131] Step S203: In response to the file processing request for the block file, each block file is processed according to the first check data segment, the block length field and the block data in the block file to obtain the file processing result.

[0132] File processing requests include file read requests, file write requests, file merge requests, and file split requests.

[0133] It should be understood that, in response to a file processing request for a block file, the embodiments of the present invention perform data verification on the block data involved in file processing based on the first verification data segment in the block file, thereby ensuring data integrity and avoiding errors in data processing results due to data anomalies. In the process of processing the block file, the byte position of the block data involved in file processing is located based on the block length field in the block file, thereby forming a data pointer to accurately reference the corresponding block data.

[0134] For example, embodiments of the present invention can verify and reference the corresponding block data in the block file based on the first verification data segment and the block length field, thereby realizing one or more combinations of file reading, file writing, file merging, and file splitting processing of the block file, and thus obtaining the corresponding file processing result. It should be noted that file reading processing generally does not change the data content of the block file. The storage management of block data in the block file can be completed by merging and / or splitting the block file.

[0135] Step S204: Based on the file processing results, and in conjunction with the second check data segment, index length field, and index data in the index files, update each index file.

[0136] It should be understood that, according to the file processing result obtained from the file processing request for the block file (e.g., the block file obtained after completing the merging and / or splitting processing), the embodiments of the present invention further combine the second check data segment, the block length field, and the index data in the index file, for example, by performing relevant operations on the index data, the second check data segment, and the block length field in the relevant index file according to the file format corresponding to the merged block file and / or the split block file, so as to realize the update processing of each index file.

[0137] The specific implementation process of the block data processing method provided by the present invention is described below with reference to the accompanying drawings:

[0138] First, it's important to note that new data is generated daily in the blockchain. Block data consists of raw data uploaded by users, primarily the blocks themselves, their headers, and transactions. This data only adds to and never removes data; once written, it's never modified again. This data characteristic makes Block File Database (BFDB) a highly suitable file processing method. The directory structure of Block File Database is as follows: Figure 3 As shown, in this structure, the block data is placed in xxx.fdb (the first original file) under the bfdb folder (the main directory of the block data), and the index information is placed in leveldb (the corresponding xxx.ldb, the second original file) under the corresponding blockdb folder. For example, Figure 4The diagram shows the structure of the first original file. The xxx.fdb file can be used to continuously store the serialized text of the blockchain (serialization here refers to a sequence ordered by block height), forming multiple serialized fdb files 40 (the first original file), including serialized texts 41 and 42, etc. (unlabeled, where 42 is the serialized text of the next block after 41). Correspondingly, the second original text can be stored as a JSON file, containing several indexed texts based on the columnarized fdb files.

[0139] Furthermore, it is possible to Figure 4 Each block in ChainMaker is preceded by checksum data. This checksum can consist of a checksum (CRC checksum, i.e., the first checksum segment) and a length character representing the block's data length (i.e., the block length field). The checksum is calculated using the CRC algorithm in ChainMaker and is 4 bytes long. The length character is a 10-byte value (enough to record the length of a 1000MB block after serialization) used to record the text length of the written block. Figure 5 As shown, after adding the verification data, block file 50 is obtained. The format of block file 50 is still an fdb file composed of multiple serialized text blocks. The difference from the first original file is that each block in the file has additional verification data before it, forming complete block data 51. 52 is the next complete block data after 51 (not labeled below). Each complete block data includes CRC verification data 53, length character 54, and serialized text 55 in sequence.

[0140] It should be noted that the second original file in this embodiment of the invention can be designed in JSON file format, for example, as shown below. Figure 6 As shown, in this embodiment of the invention, verification data can be added before each index in the JSON file. After adding the verification data, an index file 60 is obtained. The format of the index file 60 is an xxx.json file composed of multiple columnarized index texts of xxx.fdb. The difference from the second original file is that each index in the file has additional verification data 63 before it, forming complete index data 61. 62 is the next complete index data of 61 (not labeled thereafter). Each complete index data includes CRC verification data 63, length character 64, and columnarized index text of xxx.fdb 65 in sequence.

[0141] Subsequently, based on the structurally optimized block file and index file, in response to a file processing request for the block file, data verification of the block data involved in file processing is performed according to the first checksum segment in the block file. The byte positions of the block data involved in file processing are located according to the block length field in the block file, thus forming a data pointer for accurate reference to the corresponding block data. This enables one or more combinations of file reading, writing, merging, and splitting processes for the block file to obtain the corresponding file processing result. Further, based on the file processing result obtained from the file processing request for the block file (e.g., the block file obtained after merging and / or splitting), and combined with the second checksum segment, block length field, and index data in the index file, for example, according to the file format corresponding to the merged and / or split block file, relevant operations corresponding to the file format are performed on the index data, second checksum segment, and block length field in the relevant index file to achieve update processing of each index file.

[0142] In some possible implementations, the first verification data segment is used to verify the integrity of the block file; the block length field is used to characterize the field length of the block data in the block file; the second verification data segment is used to verify the integrity of the index file; the index length field is used to characterize the field length of the index information in the index file. The step of adding the block length field to each block data in the first original file in step S201 may include, but is not limited to, steps S2011 to S2012.

[0143] Step S2011: Obtain the storage size of the block data;

[0144] For example, the size of the block data in bytes can be obtained directly through relevant functions, including but not limited to the sizeof function, MongoDB and Redis functions, or by directly accessing the relational database (blockchain) of the block file to obtain the storage size of the corresponding block data. Since the number of bytes in the data is directly reflected in the storage size, and the storage size of the data can be easily obtained, obtaining the storage size of the block data can provide a parameter basis for subsequently determining the block length in bytes.

[0145] Step S2012: Calculate the block length of the block data based on the storage size, and add the block length field to the block data in the first original file.

[0146] It should be understood that a storage unit is a unit of quantity. In a particular industry, it refers to a specific quantity or standard used as a record (counting) point, and then another point is defined by a certain multiple from this point; this point is symbolized as the counting unit or storage unit. Internally, information is stored, calculated, processed, and transmitted entirely in binary format. Therefore, the data bytes can be determined first based on the storage size, and then the block length of the block data can be calculated.

[0147] For example, a 10-byte value (capable of recording the length of 1000MB of serialized data) can be reserved to record the text length of the written block; and the process of determining the block length based on the storage size of the block data, for example: if the serialized block size is 10MB, then the text size is 10485760, and the length of "10485760" is 8 bytes, which is less than the reserved 10 bytes.

[0148] In some possible implementations, when the file processing request is a file read request, step S203 may include, but is not limited to, steps S20301 to S20303.

[0149] Step S20301: In response to the file read request for the block file, obtain the file header of the block file, and read the first check data segment and the block length field from the file header in sequence;

[0150] It should be understood that the file header represents the first checksum segment and block length field added before the block data during the structural optimization process of the first original file. The first checksum segment is a check value calculated from the original block data using a relevant checksum algorithm (e.g., CRC). This checksum is typically not modified and is used as a fixed value for comparison in subsequent data verification processes. Since the block file in this embodiment of the invention comprises several blocks of data, the block length field can assist in accurately locating the field positions of the processed data during subsequent processing of related block data.

[0151] For example, with (such as) Figure 3The file shown uses the xxx.fdb format structure for the block file and the CRC checksum algorithm (the resulting checksum is 4 bytes) as an example. The block length field can reserve 10 bytes (enough to record the length of the serialized data of a 1000MB block). That is, the file header is a total of 14 bytes (including a 4-byte first checksum data segment and a 10-byte block length field). The xxx.fdb contains several block data with file headers. The process of reading the file header of any block data can be as follows: first, obtain the file header from the beginning of the block data, and then read the 4-byte checksum (the checksum calculated by CRC, i.e., the first checksum data segment) and the 10-byte "block length" from the file header.

[0152] Step S20302: Based on the block length field, read data of the corresponding length from the block file to obtain block data;

[0153] It should be understood that, since the block file of the embodiments of the present invention includes the structural composition characteristics of several block data, in order to accurately obtain the corresponding block data and avoid data extraction errors caused by the mixing of data from various blocks during the block data acquisition process, the block length field can assist subsequent related block data processing in accurately locating the field position of the processed data.

[0154] For example, with (such as) Figure 3 Taking the block file shown as an example, which uses the xxx.fdb format structure, it should be noted that, in order to facilitate the accurate acquisition of block data, the "block length" can be converted into a numerical value, which is the length data of the block data. Then, the text of a specified length can be read according to the length data to obtain the text (i.e., block data) stored in xxx.fdb after block serialization.

[0155] Step S20303: Calculate the verification information of the block data, compare the verification information with the first verification data segment, determine the integrity of the block data, and continue reading all block data in the block file until the reading is completed.

[0156] It should be understood that, to verify the integrity of block data, verification information is first calculated based on the read block data. The specific calculation process is consistent with the process of calculating the first checksum segment based on each block data in the first original file. However, this step calculates the block data that has been reread. If data corruption occurs during data storage, the resulting block data may not be identical to the corresponding block data in the first original file. Therefore, the verification information needs to be compared with the first checksum segment to determine the integrity of the block data. This process of reading and verifying the integrity of each block data in the block file is repeated until all block data in the block file has been read.

[0157] For example, with (such as) Figure 3 Taking a file with a block structure of xxx.fdb and a verification algorithm of CRC as an example, the block data read is the text stored in xxx.fdb after block serialization. The checksum (the check value of the text, which represents the verification information) is calculated by CRC based on this text, and then compared with the checksum (the first verification data segment) read in step S20301. If the two check values ​​are the same, it means that the data is complete. The serialized block text in xxx.fdb is read in turn and the data integrity is determined until all block text in xxx.fdb is read.

[0158] In some possible implementations, the method of the present invention may also include, but is not limited to, step S20304.

[0159] Step S20304: When reading block data from the block file results in erroneous data, terminate the reading of the block file and discard all content following the erroneous data.

[0160] It should be understood that block data may be corrupted during storage. Causes of corruption can include: unforeseen circumstances, such as natural disasters like floods, fires, lightning strikes, and earthquakes that damage computer systems, leading to data corruption or complete loss; user error; hardware failures of computer equipment, including aging or failure of storage media; and data corruption caused by virus infection. Corrupted or erroneous block data is unusable. Therefore, when reading erroneous block data from a block file, the reading process is terminated, and all content following the erroneous data is discarded to prevent errors in further calculations and processing of related block data. Since a block file contains several blocks, discarding subsequent data after the erroneous data point aims to preserve as much complete preceding data as possible, improving data utilization and preventing subsequent erroneous data from contaminating the overall data.

[0161] For example, with (such as) Figure 3 Taking a file with an xxx.fdb format as an example, the process involves sequentially reading the header of each block in xxx.fdb, determining the length of the current block based on the block length in the header, calculating the verification value of the block, and comparing it with the first suggested data segment in the header to verify the integrity of the current block. If an error is found in any block during the reading process, it indicates that subsequent data is corrupted, the reading process terminates, and the remaining xxx.fdb text is discarded, while the previously read text can be saved separately.

[0162] In some possible implementations, when the file processing request is a file write request, step S203 may include, but is not limited to, steps S20305 to S20306.

[0163] Step S20305: In response to the file write request for the block file, the first verification data segment, the block length field, and the block data are written to the block file in sequence.

[0164] It should be understood that when new block data needs to be uploaded to the blockchain (e.g., from a blockchain user), in response to a file write request to the block file, where the file write request can be generated based on the blockchain user's data upload operation, the block data is written to the block file after structural optimization. Correspondingly, after obtaining the block data to be written but before writing it to the block file, in order to ensure that the data format of the newly written block data corresponds to the format characteristics of the block file, relevant verification and byte length calculations can be performed on the block data to be written, to obtain its corresponding first verification data segment and block length field, and then the first verification data segment, block length field, and block data are written to the block file in sequence.

[0165] Step S20306: Write the index information corresponding to each block of data into the index file, and after writing all the block data, add an end marker to the index file. The end marker is used to indicate the last block of data written into the block file.

[0166] It should be understood that after the operation of writing new block data to the block file is completed, the index information corresponding to the block data is then written to the index file. At this time, the index information written is in the index file after structural optimization. Correspondingly, after obtaining the index information corresponding to the written block data and before writing it to the block file, in order to ensure that the data format of the newly written index information corresponds to the format characteristics of the index file, relevant verification and byte length calculations can be performed on the written index information to obtain its corresponding second verification data segment and index length field. Then, the second verification data segment, index length field, and index information are written to the index file in sequence. Among them, block data is often written in batches. In order to prevent data corruption caused by failure or crash during the writing process, after a block of data is completely written, an end marker is added to the index file. When the subsequent block of data is completely written, an end marker is added again (updating or overwriting the original end marker). The end marker always represents the last block of data written to the block file.

[0167] For example, when the file processing request is a file write request, in order to (e.g.) Figure 3Taking a block file with an xxx.fdb format structure and a CRC checksum algorithm (the checksum is obtained as the first and second checksum data segments) as an example, in some specific embodiments, step S203 may include the following steps: When writing a block to the xxx.fdb file, first write the checksum, then write the "block length", and finally write the block text data; after writing xxx.fdb, write the index data corresponding to the block to the KVDB (i.e., the index file). To prevent failures in writing the index data (e.g., KVDB crash), after all the indexes are written, write a separate lastsavepoint identifier to record the block height corresponding to the index. The order of writing blocks and indexes is: block (N-1) --> block index (N-1) --> block (N) --> block index (N) --> block (N+1) --> block index (N+1). This embodiment of the invention uses the aforementioned block and index writing order as an example. Under this example, this embodiment of the invention can ensure that when any step in the block data processing flow is abnormally interrupted, restarting the processing flow guarantees that the data will not be corrupted or lost. Following the one-to-one writing order of blocks and block indexes facilitates direct continuation of writing from the interrupted position when an abnormal interruption occurs, ensuring the integrity of the data before the interruption, the one-to-one correspondence between blocks and block indexes, and guaranteeing that the preceding block data can be used normally based on the index during the interruption. Otherwise, if all blocks are written to the database at once, and a processing interruption occurs, the written blocks will be unusable because the corresponding index information is not stored in order. They can only be used after all blocks and indexes have been written. Therefore, the planning of the block data writing order and index data writing order in this embodiment of the invention can greatly improve the processing efficiency when the system is interrupted.

[0168] In some possible implementations, the method of the present invention may also include, but is not limited to, step S20307.

[0169] Step S20307: When a write error occurs while writing block data to the block file, the write process to the block file is terminated, and when the write process is restarted, the end marker is read, and the block data is rewritten starting from the last block data.

[0170] It should be understood that when a write error occurs while writing block data to the block file, in order to avoid rewriting all block data and increasing the amount of unnecessary work, the write process to the block file is terminated to ensure the integrity of the block data that has already been written. Then, when the write process is restarted, the end identifier is read, and the last block data written in the block file represented by the end identifier is determined. Then, the block data is rewritten starting from the last block data.

[0171] For example, with (such as) Figure 3 The following example illustrates the overall processing steps in response to a file write request, using a file with a block file structure in the format of xxx.fdb and a CRC checksum algorithm (the checksum value is obtained from the first and second check data segments):

[0172] When writing a block to the xxx.fdb file, first write the checksum, then write the "block length", and finally write the block text data. If an error occurs during the writing process, the writing will be terminated and the node will be terminated. When writing again, the node needs to be restarted. At this time, you can follow steps S20301 to S20303 to read the file to load the block data.

[0173] Assumption: After writing block N, the node server experiences a power outage while writing block (N+1), resulting in incomplete writing of block (N+1), such as writing 0.5 blocks. When the node restarts and loads the latest block from xxx.fdb, it will discard (N+1) blocks and only recognize the blocks it stored up to block height N.

[0174] Furthermore, after writing to xxx.fdb, the index data corresponding to the block is written to KVDB (index file). To prevent failures during index data writing (e.g., KVDB crash), a separate lastsavepoint identifier is written after all indexes are written, recording the block height corresponding to the index being written. The order of block and index writing is: Block (N-1) --> Block Index (N-1) --> Block (N) --> Block Index (N) --> Block (N+1) --> Block Index (N+1).

[0175] Assumption: After writing block and index data of height N, the database crashes when writing "block (N+1)" and then "block index (N+1)".

[0176] The node reads the value returned by lastsavepoint in KVDB upon restart;

[0177] If N+1 is returned, it means that the data storage of block N+1 is complete, and the process of writing to N+2 can continue.

[0178] If N is returned, it means that the block N+1 data is incomplete. The storage needs to recalculate the index data based on the block data of height (N+1) in xxx.fdb, and write the "block index (N+1)" data back into KVDB. Due to the characteristics of KVDB, if data has been written before, it will be overwritten by the new data. There is no need to worry about data corruption caused by multiple writes.

[0179] In some possible implementations, when the file processing request is a file merging request, step S203 may include, but is not limited to, steps S20308 to S20312.

[0180] Step S20308: In response to the file merging request for the block files, obtain several block files that are temporally adjacent.

[0181] It should be understood that in certain scenarios, such as when restoring nodes after block archiving (archiving refers to moving inactive but necessary data to another storage system to free up node main storage and improve system performance; after data is archived, it can be restored from the archived data if needed), the blocks restored to the node by blockchain users may be discontinuous (even in this case, they will still be kept as block files, with each batch of blocks increasing sequentially in height placed in a block file), or the order may not be pushed according to the order in which the blocks are stored in the block file (for example, if the blocks are written to the block file in reverse order during restoration, each block will be saved as a separate block file). Therefore, after receiving the blocks, the node needs to organize these block data and merge them according to the preset storage rules and the size of a single block file. Therefore, by triggering a file merge request for relevant block files at the relevant specific time node (such as the node during restoration after archiving), several temporally adjacent block files are obtained as the data basis for subsequent merging processing.

[0182] Step S20309: Using the first block file of several adjacent block files as the merge file, merge the other block files into the merge file in sequence.

[0183] Each block of data in the merged file includes a first checksum data segment and a block length field.

[0184] It should be understood that since block data is arranged by height, the block data in adjacent block files is contiguous. That is, in two adjacent block files, the height of the last block in the preceding block file is one greater than the height of the first block in the following block file. Therefore, merging files using several adjacent block files, and using the first block file of these adjacent block files as the merge file, allows for the sequential merging of block data from other block files into the merge file. This avoids excessive file manipulation; subsequent block files are simply merged into the merge file sequentially, making file merging convenient.

[0185] Step S20310: Based on the information of the new block data in the merged file, update the index file of the first block file to obtain the index file corresponding to the merged file.

[0186] It should be understood that after merging several adjacent block files, the index files of the corresponding block files may need to be updated based on the merged file structure so that subsequent calls to related block data based on the index files can accurately guide to the corresponding block files. Since each index file includes several index information (data) corresponding to the block data in the corresponding block files, updating the index files can be done similarly to the block file merging process. First, the update is based on the adjacent index files corresponding to the adjacent block files being merged. Then, the update file is based on the index file of the first block file among the adjacent block files to be merged. Next, the index information from the index files of the other block files is merged into the index file sequentially. Finally, the index files corresponding to subsequent block files are merged into the update file in turn to obtain the index file corresponding to the merged file.

[0187] Step S20311: Store the filenames of each block file merged into the merged file in the target file.

[0188] It should be understood that by storing the filenames of the various block files merged into the merged file in the target file, the filenames of the various block files merged into the merged file are recorded as file pointers for subsequent processing of the relevant block files, so as to quickly locate the corresponding block files.

[0189] Step S20312: Based on the filename of the target file, delete the corresponding block files and index files from the temporary folder.

[0190] The temporary folder is used to store files during the processing.

[0191] It should be understood that, to avoid errors in the file merging process causing damage to the root file of the block files, a temporary folder can be created during the merging process to store intermediate files, such as the block files to be merged into the merged file, and the index files used to update the corresponding index files of the merged file (i.e., the index files corresponding to the block files being merged). Furthermore, the merging of block files and the updating of index files can be performed based on the relevant block files and index files in the temporary folder, avoiding file loss or corruption caused by arbitrary modifications to the root file. Finally, based on the filename of the target file, the corresponding block files and index files are deleted from the temporary folder.

[0192] It should also be noted that in some embodiments, file merging and index updating are based on transferring block data from relevant block files to the merged file and transferring index information from relevant index files to the index file corresponding to the merged file. The merged block files and their corresponding index files still exist in the root directory. In this case, the block files and index files with corresponding filenames in the root directory can also be deleted by using the filenames in the target file to avoid data redundancy.

[0193] In some possible implementations, step S20309 may include, but is not limited to, steps S203091 to S203092.

[0194] Step S203091: Based on the filename of each block file, parse the block height of each block file, and determine that each block file is a temporally adjacent block file based on the block height of each block file.

[0195] It should be understood that since the block data is arranged according to height, the block files can be determined to be temporally adjacent based on their block heights. That is, in two adjacent block files, the height of the last block data in the preceding block file is 1 greater than the height of the first block data in the following block file. In some embodiments, the filename of each block file represents the height of the first block data in that block file. Since a block file usually contains several blocks of data, the filenames of the block files are usually not consecutive. For example, if the first block file contains blocks of data with heights of 1 to 5, then the filename of the first block file is xxx1; if the second block file contains blocks of data with heights of 6 to 9, then the filename of the second block file is xxx6, and so on. Therefore, the block heights of each block file are parsed, and the temporally adjacent block files are determined based on these block heights. When two block files have no intermediate filename, they can be considered to be adjacent filenames. The file order can be further determined based on the block height determined by the filename. By analogy, the sequence of all block files can be determined.

[0196] Step S203092: Take the first block file of each block file with adjacent time sequences as the merge file, and merge the other block files into the merge file in sequence.

[0197] Understandably, based on the time sequence (height sequence) of the block files determined in the aforementioned steps, multiple block files with adjacent time sequences are then merged. Specifically, the first block file among several block files with adjacent time sequences to be merged is used as the merge file. Then, the block data in each of the other block files can be merged into the merge file in sequence. This avoids excessive file manipulation. Subsequent block files are simply merged into the merge file in sequence, making file merging convenient.

[0198] For example, with (such as) Figure 3 The example shown uses a block file with the xxx.fdb format structure. Here, xxx.fdb supports merging, meaning multiple (two or more) xxx.fdb files can be merged into a larger xxx.fdb. The purpose of merging to obtain a larger xxx.fdb is to address specific needs in certain scenarios, such as when restoring data after node block archiving (archiving refers to moving inactive but necessary data to another storage system to free up primary storage and improve system performance. After data is archived, if it needs to be used again, the relevant data can be restored from the archived data). The blocks restored to the node may be discontinuous (in which case they will still be kept as xxx.fdb files, with each batch of blocks increasing in height sequentially placed in a single xxx.fdb file), or the order may not follow the order in which the blocks are stored in xxx.fdb (for example, if the blocks are written to xxx.fdb in reverse order during restoration, each block will be saved as a separate xxx.fdb file). Therefore, after receiving the blocks, the node will organize these block data, merging and organizing them according to the storage rules of xxx.fdb and the size of a single xxx.fdb file to resolve the issue of scattered xxx.fdb files in bfdb.

[0199] In some specific embodiments, such as Figure 7 As shown, you can see the effect of merging 0000000000000000006.fdb and 00000000000000000010.fdb into 00000000000000000001.fdb. Then, delete the merged 00000000000000000006.fdb and 0000000000000000010.fdb. Specifically, the merging process for xxx.fdb files can include:

[0200] 1. Create a merge folder (i.e., a temporary folder) under the bfdb folder (i.e., the main directory of block data);

[0201] 2. Confirm that the content of the merged file is continuous: parse the height of the file name 00000000000000000006.fdb: 6, and obtain the index information of the two blocks with heights of 4 and 5 from KVDB (index file). Height 5 is the height of the first block in 00000000000000000006.fdb (in ChainMaker, the height of bfdb is 1 greater than the block height, so 5 = 6 - 1), and FileName is 000000000000000000006; then, get the block with height 4 (4 = 5 - 1), which is the height of the last block in 00000000000000000001.fdb, and FileName is 00000000000000000001. Therefore, 00000000000000000001.fdb and 00000000000000000006.fdb are two consecutive files. If the FileName in the index of the two consecutive blocks is different, it means that the xxx.fdb files containing the two consecutive blocks are consecutive.

[0202] 3. Merging involves sequentially merging the contents of subsequent xxx.fdb files into the first xxx.fdb file. Therefore, the merged filename will be the same as the first xxx.fdb file. Figure 7 As shown;

[0203] 4. Copy the first xxx.fdb file (e.g., 0000000000000000001.fdb) to the merge folder;

[0204] 5. Parse the contents of the next xxx.fdb (e.g., 0000000000000000002.fdb), recalculate the index information of the xxx.fdb and store it in an index file named after the first xxx.fdb file (e.g., 0000000000000000001.json). Then, append the contents of the xxx.fdb file directly to merge / 000000000000000000001.fdb.

[0205] 6. If there are other xxx.fdb files that need to be merged, merge them into the first xxx.fdb file in sequence, following the processing method in step 5.

[0206] 7. xxx.json file structure: When adding indexes to the xxx.json file, the index data for subsequent blocks must also refer to the block addition method in xxx.fdb;

[0207] 8. After adding the index, add the filenames to xxx.file (e.g., 00000000000000000001.file, i.e., the target file). This adds the filenames of all merged files to the file so that individual merged files can be deleted from the root directory, resulting in a file like this. Figure 8 The state shown:

[0208] 8.1 The merge / 00000000000000000001.fdb file stores the final state of the merged blocks;

[0209] 8.2 The merge / 00000000000000000001.json file contains the index data of the blocks in the final state after the merger;

[0210] 8.3 The file merge / 00000000000000000001.file stores the filenames related to the merge, with each filename occupying one line. The first filename is not stored. The following is the content of merge / 000000000000000000001.file: 00000000000000000006;

[0212] 0000000000000000010.

[0213] 9. Overwrite bfdb / 00000000000000000001.fdb with merge / 0000000000000000001.fdb, and then delete merge / 0000000000000000000001.fdb;

[0214] 10. Write the data in merge / 00000000000000000001.json to the KVDB that stores the block index (e.g., Figure 3 After writing (e.g., leveldb), delete merge / 000000000000000000001.json. If the index writing process is interrupted, the old index data that has not been written to KVDB can still be read from the old xxx.fdb (e.g., ...). Figure 8 (bfdb / 0000000000000000006.fdb and bfdb / 0000000000000000010.fdb);

[0215] 11. Read the data in merge / 000000000000000000001.file, delete the file with the corresponding filename in the bfdb folder, and then delete the merge / 000000000000000000001.file file; if the file under bfdb recorded in merge / 000000000000000000001.file does not exist, you can directly delete the merge / 0000000000000000000001.file file.

[0216] In some possible implementations, when the file processing request is a file merging request, step S203 may also include, but is not limited to, steps S20313 to S20317.

[0217] Step S20313: When a merge error occurs while merging block files, check the file types present in the temporary folder:

[0218] It is understandable that the merge process can be terminated at any abnormal step to ensure data integrity to the greatest extent possible. Therefore, when a merge error occurs during the merging of block files, the file types existing in the temporary folder should be checked, and adaptive processing, such as steps S20314 to S20317, can be performed based on the file types present in the temporary folder. It should be noted that the merge error here refers to the merge error that occurs when the merge process is forcibly interrupted due to reasons such as blockchain node errors or server crashes and restarts. In some embodiments, when an error (procedural or logical, etc.) occurs during the merge, the merge is terminated directly, the error is reported to the administrator, the merged files and their corresponding index files generated during the merge process are deleted, the temporary folder is further emptied, and the merge environment is reset. The merge process is then restarted after the administrator corrects the relevant errors.

[0219] Step S20314: When the temporary folder contains a merged file, an index file, and a target file, repair the merged file, the index file, and the target file to complete the file merging of the block file.

[0220] For example, such as Figure 9 As shown, when the temporary folder contains the merge file, index file, and target file simultaneously, it indicates that the merge has begun and will terminate after a period of time. After reading the index file, the following steps can be performed:

[0221] 1. First, determine if there is an error in parsing the index file corresponding to the merged file;

[0222] 2. If a parsing error occurs, discard the incomplete index data in the index file and proceed to the next step; otherwise, if the parsing is successful, proceed directly to the next step.

[0223] 3. Verify the contents of the merged file based on the information in the index file, delete the data in the merged file that does not store the index; and find the filenames of the files involved in the merged file based on the index file, and fill in the missing filenames in the target file.

[0224] 4. The data structure has been repaired; continue the unfinished merging work.

[0225] 5. After the merge is complete, clear the temporary folder and reset the merge environment.

[0226] Step S20315: When only merged files exist in the temporary folder, delete the merged files to complete the file merging of the block files;

[0227] For example, if the temporary folder only contains the merged files, it indicates that the merge process terminated just as it began. The merged files can then be deleted (and the merge process can be restarted until completion), and the temporary folder emptied to reset the merge environment. Alternatively, the administrator can resend the file merge request, and the merge process can be restarted in response to the request to perform the relevant file merge processing.

[0228] Step S20316: When there is no merged file in the temporary folder, but there is an index file and a target file, write the data in the index file to the original folder, and delete the block files and index files with the corresponding file names from the temporary folder according to the file name of the target file, thus completing the file merging of the block files;

[0229] For example, if the merged file is not present in the temporary folder, but the index file and target file exist, it indicates that the merging of the files has been completed, and the merged file in the temporary folder has been deleted. Therefore, it is necessary to update the index file. This can be done by writing the data from the index file to the original folder, and deleting the merged file and its index file in the root directory. Based on the target file's filename, the corresponding block files and index files can be deleted from the temporary folder, completing the merging of the block files. Finally, the temporary folder is cleared, and the merging environment is reset.

[0230] Step S20317: When the temporary folder contains only the target file, delete the corresponding block files and index files from the temporary folder according to the file name of the target file, and complete the file merging of the block files.

[0231] For example, if neither the merge file nor the index file exists in the temporary folder, and only the target file exists, it indicates that the merging of the files has been completed. Simultaneously, the merge file in the temporary folder has been deleted, and the contents of the index file have been written to the original folder. Therefore, based on the target file's filename, it is only necessary to delete the corresponding block files and index files from the temporary folder to complete the merging of the block files. Finally, the temporary folder is cleared, and the merging environment is reset.

[0232] For example, in some specific embodiments, as shown in (e.g.) Figure 3 (As shown) The block file uses the xxx.fdb format structure as an example. Figure 10 As shown, if a merge error occurs during the file merging process, the following steps should be taken following the previously mentioned xxx.fdb merging process:

[0233] 1. If an error (procedural or logical, etc.) occurs during the merger, terminate the merger immediately, report the error to the management personnel, and then proceed to step 7; otherwise, proceed to the next step.

[0234] 2. When a merge error occurs during the merge process due to reasons such as a blockchain node error or a server crash and restart, check the file types in the merge folder (i.e., the temporary folder), and then perform adaptive processing based on the file types in the merge folder, as in any of steps 3 to 6.

[0235] 3. When the merge folder contains xxx.fdb (the merge file), xxx.json (the index file), and xxx.file (the target file), it indicates that the merge has started and has been running for some time before stopping. You can then proceed with the following steps:

[0236] 3.1 First, determine if there is an error in parsing xxx.json;

[0237] 3.2 If parsing xxx.json fails, discard the incomplete index data in xxx.json and proceed to the next step; otherwise, if parsing is successful, proceed directly to the next step.

[0238] 3.3. Verify the contents of xxx.fdb based on the information in xxx.json, delete the data in xxx.fdb that does not store indexes; and find the filenames related to xxx.fdb based on xxx.json, and complete the missing filenames in xxx.file.

[0239] 3.4 The data structure has been repaired. Proceed to step 5 of the aforementioned xxx.fdb merging process to continue the previously unfinished merging work.

[0240] 3.5 After the merge is complete, proceed to step 7.

[0241] 4. If only xxx.fdb exists in the merge folder, it means that the merge has just started and has stopped. You can then delete xxx.fdb until the merge is complete and proceed to step 7.

[0242] 5. If xxx.fdb does not exist in the merge folder, but xxx.json and xxx.file exist, it indicates that the xxx.fdb file has been merged. Then, jump to step 10 of the aforementioned xxx.fdb merging process and start executing again; until the merge is completed, jump to the subsequent step 7.

[0243] 6. When xxx.fdb and xxx.json do not exist in the merge folder, and only xxx.file exists, it indicates that the content of xxx.json has been written to KVDB (shown as kvdb). Then, the process jumps to step 11 of the aforementioned xxx.fdb merging process and starts again. Until the merging is completed, the process jumps to step 7.

[0244] 7. Clear the merge file and reset the merge environment.

[0245] In some possible implementations, when the file processing request is a file splitting request, step S203 may include, but is not limited to, steps S20318 to S20320.

[0246] Step S20318: In response to the file segmentation request for the block file, determine the segmentation height and the number of files to be segmented into the block file;

[0247] It should be understood that in certain scenarios, such as when archiving blocks on a node, the archive height specified by the blockchain user may be in the middle of a block file. For example, a block file may contain blocks 10, 11, 12, 13, 14, and 15, and the blockchain user wants to archive blocks with a height up to 13. In this case, the corresponding block file needs to be split before the archiving operation. Therefore, to facilitate the archiving operation, the block file split height and the number of split files are determined in response to the block file splitting request. Then, the block requiring the archiving height can be located using the determined split height and the number of split files, facilitating subsequent splitting processing.

[0248] Step S20319: Based on the segmentation height and the number of segmented files, divide the block file into a corresponding number of sub-files;

[0249] Each block of data in the sub-file includes a first checksum data segment and a block length field;

[0250] It should be understood that since the block data in the block file is arranged according to height, when a block of a certain target height is in the block file, it is necessary to determine the split point of the block file based on the split height and the number of split files. Then, based on the determined split point, the block file is split into a corresponding number of sub-files to ensure that the block of the target height is at the beginning or end of the split sub-files after splitting, so as to facilitate the archiving operation of blockchain users based on the relevant target height blocks.

[0251] Step S20320: When a sub-file is obtained from the split, configure the index file corresponding to the sub-file.

[0252] It should be understood that, to facilitate archiving operations and referencing the segmented block sub-files, an index file needs to be configured for each sub-file after the segmentation operation is completed. Specifically, the index file for each sub-file can be configured by performing the same segmentation operation on the index file corresponding to the segmented block file, thereby obtaining the corresponding index file for each sub-file; alternatively, the index file for each sub-file can be directly obtained by calculating the index of each segmented sub-file, thus overriding the index file corresponding to the segmented block file.

[0253] It's also worth noting that the file splitting process can create temporary folders, similar to those used in file merging, to store intermediate files during processing, such as the individual sub-files and their corresponding index files. This prevents file loss or corruption caused by arbitrary modifications to the root file. Furthermore, this temporary folder serves as a reference, allowing for real-time monitoring of the intermediate file status during the splitting process, and enabling the measurement of the splitting process's status or the determination of splitting errors based on the status of these intermediate files.

[0254] In some possible implementations, step S20318 may include at least one of the following:

[0255] Identify the overall block height of the block file and determine the height of each segment from the overall block height based on the number of segmented files parsed in the file segmentation request;

[0256] Alternatively, calculate the data size of the block file, and determine the number of files to be split and the block file split height based on the size of the file to be split parsed in the file splitting request.

[0257] In some possible implementations, the method of the present invention may also include, but is not limited to, steps S20320 to S20321.

[0258] Step S20320: In response to the file splitting request for the block file, record the file processing status of the block file as the file splitting status;

[0259] It should be understood that when a block file is being split, the block data and its corresponding index information within that block file are undergoing simultaneous processing steps. If other operations that modify the block data and index information are performed concurrently (e.g., adding, deleting, or modifying data), it can easily cause confusion in the related data, leading to the failure of the splitting and related modification processes, and ultimately data corruption. Therefore, in response to a file splitting request for a block file, the file processing status of the block file is recorded as a file splitting status. Specifically, during the splitting process, the relevant block files are marked with a status to ensure that the data in the current block file only supports splitting processing, and other permissions for related modification operations that require scheduling of the block file are temporarily restricted.

[0260] Step S20321: When the file segmentation process of the block file is completed, change the file processing status to the file writing status.

[0261] It should be understood that when the file segmentation process of a block file is completed, the file processing status is changed to a file write status, the relevant block file undergoes a status transition, and permissions for other related change operations that require scheduling of that block file are re-enabled. This embodiment of the invention, by using two nodes—segmentation request and completion of segmentation processing—to mark and transition the status of the corresponding block file, ensures that data is not repeatedly accessed during the file segmentation process, thus avoiding data corruption or loss.

[0262] For example, with (such as) Figure 3 Taking a file with the xxx.fdb format structure as an example (shown), the xxx.fdb supports splitting, that is, splitting a single xxx.fdb into multiple smaller xxx.fdb files (the block text in the split xxx.fdb corresponds to...).

[0263] Distribute the data into smaller xxx.fdb files. The purpose of splitting the data into multiple smaller xxx.fdb files is to address specific needs in certain scenarios. For example, when archiving blocks on a node, if the blockchain user specifies an archive height in the middle of a block file (e.g., a block file contains blocks 10, 11, 12, 13, 14, and 15), and the user wants to archive blocks up to height 13, then the corresponding block file needs to be split before archiving. This splitting process facilitates accurate archiving.

[0264] In some specific embodiments, the xxx.fdb segmentation process may include:

[0265] 1. Create a folder named spilt (similar to a temporary folder) under the bfdb folder;

[0266] 2. Calculate the block height of xxx.fdb according to certain rules:

[0267] 2.1 For example, if 00000000000000000001.fdb contains block data with block heights from 0 to 13, and it is divided into two parts according to the block height, the dividing point is block height 7;

[0268] 2.2 If the partitioning is based on the size of the data within xxx.fdb, then the size of each block in xxx.fdb is calculated, and the partitioning is performed based on the total size of consecutive blocks. If the total size of consecutive blocks is greater than or equal to 64MB, then the partition is performed once. For example, 00000000000000000001.fdb contains blocks 0-13, with a total size of 192MB. The partitioning points are calculated to be 4 and 8. Figure 11 As shown, this splits 0000000000000000001.fdb into three files, each 64MB: 000000000000000000001.fdb, 00000000000000000006.fdb, and 00000000000000000010.fdb.

[0269] Taking the splitting of bfdb / 00000000000000000001.fdb as an example, the height points of the split blocks are 4 and 8, and finally it is split into three files. The final state is shown in Table 1.

[0270] Table 1

[0271] Filename after splitting Contains blocks File size 0000000000000000001.fdb 0、1、2、3、4 64MB 0000000000000000006.fdb 5、6、7、8 64MB 00000000000000000010.fdb 9、10、11、12、13 64MB

[0272] 2.3 Create the file split / 00000000000000000001.state (a marker file that records the file processing status), and write the word "spliting" into it to record the splitting status;

[0273] 3. Read the bfdb / xxx.fdb file, create the split / 00000000000000000001.fdb file, and sequentially write the block data from the start (0) to the first split point (4);

[0274] 4. Create the file split / 00000000000000000006.fdb and continue writing the block data from the first split point (5 = 4 + 1) to the second split point (8). The data structure of split / 00000000000000000006.fdb is similar to... Figure 5 Consistent with the previous method, and immediately after each block is written, the corresponding index is written to split / 000000000000000000006.json, with the index file structure being consistent with the previous method. Figure 6 Consistent. Write the corresponding block data to split / 000000000000000000010.fdb and index data to split / 000000000000000000010.json according to the same rules;

[0275] 5. For example Figure 12 Split bfdb / 0000000000000000001.fdb into the split folder;

[0276] 6. Write the word "writing" to split / 000000000000000000001.state, overwriting the original "spliting", indicating that the splitting process has moved from splitting to writing data.

[0277] 7. Move split / 00000000000000000010.fdb to bfdb / 00000000000000000010.fdb, and then write all the index information in split / 00000000000000000010.json to KVDB (e.g., leveldb);

[0278] 8. When moving xxx.fdb, to prevent data loss or corruption during the move, the "move" here adopts the method of copying first, and then deleting the source data after the copy is successful;

[0279] 9. Following the method in step 6, first process split / 00000000000000000006.fdb and split / 00000000000000000006.json, then process split / 00000000000000000001.fdb and split / 00000000000000000001.json (split / 0000000000000000000001.fdb will overwrite bfdb / 000000000000000000001.fdb), until all files in split are processed in reverse order;

[0280] 10. After processing, delete split / 00000000000000000001.state. Finally, split will become an empty folder, and bfdb will become... Figure 11 The status indicates that xxx.fdb has been successfully split.

[0281] In some possible implementations, when the file processing request is a file splitting request, step S203 may also include, but is not limited to, steps S20322 to S20323.

[0282] Step S20322: When a splitting error occurs during file splitting of the block file, stop the file splitting operation and delete the generated sub-files and the index files corresponding to the sub-files;

[0283] Understandably, the segmentation process can be terminated at any abnormal step to ensure data integrity to the greatest extent possible. Therefore, in some embodiments, when an error (procedural or logical, etc.) occurs during segmentation, the segmentation is terminated directly, an error report is sent to the administrator, the sub-files generated during the segmentation process and their corresponding index files are deleted, the temporary folder used to store intermediate files generated during the segmentation process is further emptied, and the merging environment is reset. The segmentation process is then restarted after the administrator corrects the relevant errors.

[0284] Step S20323: When a blockchain network failure occurs during the file splitting of the block file, data repair is performed based on the file processing status of the block file until the file splitting process of the block file is completed.

[0285] It should be understood that when a blockchain network failure occurs during block file splitting, the first step is to determine the file processing status of the block file. Then, adaptive processing, such as steps S203231 to S203233 and steps S203234 to S203235, can be performed based on the file processing status. It should be noted that splitting errors caused by blockchain network failures refer to splitting errors that occur when the splitting process is forcibly interrupted due to reasons such as blockchain node errors or server crashes followed by restarts.

[0286] In some possible implementations, when the file processing state of the block file is the file split state, step S20323 may include, but is not limited to, steps S203231 to S203233.

[0287] Step S203231: When each sub-file obtained from the block file segmentation exists in pairs with the corresponding index file, repair the sub-files according to the index file and restart the file segmentation process of the block file.

[0288] It should be understood that, such as Figure 13 As shown, when the file processing status of the block file is in the file splitting state, and when each sub-file obtained from the block file splitting exists in pairs with its corresponding index file, it indicates that the splitting has started and terminated after a period of time. After reading the index file, the following steps can be performed:

[0289] 1. First, determine if there is an error in parsing the index file;

[0290] 2. If a parsing error occurs, discard the incomplete index data in the index file and proceed to the next step; otherwise, if the parsing is successful, proceed directly to the next step.

[0291] 3. Verify the contents of the sub-files based on the information in the index file, and delete any data in the sub-files that does not contain the index;

[0292] 4. The data structure has been repaired; continue the previously unfinished partitioning work.

[0293] 5. After completing the split, clear the temporary folder and reset the split environment.

[0294] Step S203232: When the block file is split into pairs of sub-files and index files, and a single sub-file is obtained, delete the single sub-file and restart the file splitting process of the block file.

[0295] For example, when the block file's processing status is in the file splitting state, and the block file is split into partially paired sub-files and an index file, as well as some standalone sub-files, it indicates that the partial splitting has terminated at the very beginning. This can be resolved by deleting the standalone sub-files (and restarting the splitting process until completion), clearing the temporary folder, and resetting the splitting environment. Alternatively, the administrator can resend the file splitting request, and in response, restart the splitting process at the point where the block file splitting terminated. That is, the partially paired sub-files and index file represent the completed splitting portion, while the subsequent splitting process begins on the unfinished portion.

[0296] Step S203233: When the block file is split into only one sub-file, directly restart the file splitting process of the block file.

[0297] For example, when the block file's processing status is in the file splitting state, and there is only one sub-file in the temporary folder, it indicates that the splitting process has terminated just as it began. This can be resolved by deleting that single file (and restarting the splitting process until completion), emptying the temporary folder, and resetting the splitting environment. Alternatively, the administrator can resend the file splitting request, and the splitting process can be restarted in response to the request, starting the relevant file splitting process from the beginning.

[0298] In some possible implementations, when the file processing status of the block file is in the file write state, step S20323 may include, but is not limited to, steps S203234 to S203235.

[0299] Step S203234: When each sub-file obtained from the block file segmentation exists in pairs with the corresponding index file, the sub-files are transferred to the block management database, and the file segmentation process of the block file is restarted.

[0300] The block management database is used to store the processed files.

[0301] It should be understood that when the block file is in the file writing state, and when each sub-file obtained from the block file is paired with its corresponding index file, it indicates that the block file has been completely split and the sub-files of the split processing result have been obtained. However, the sub-files are at this time as intermediate files and have not yet been written into the root directory of the block file. Then, the split sub-files can be written into the root directory of the block file (overwriting or updating the split block files in the root directory of the block file), and the index files corresponding to each sub-file can be written into the root directory of the index file (overwriting or updating the index files corresponding to the split block files in the root directory of the block file), until all intermediate files in the temporary folder are written into their corresponding root directories, and the temporary folder is cleared and the splitting environment is reset.

[0302] Step S203235: When the block file is split into pairs of sub-files and index files, and a separate index file is obtained, the index file is written to the index database, and the file splitting process of the block file is restarted.

[0303] The index database is used to store index information for block data.

[0304] It should be understood that when the block file is in the file writing state, and when the block file is split into partially paired sub-files and index files, and also into a standalone index file, it indicates that the block file has been completely split, and the sub-files resulting from the splitting process have been written into the root directory of the block file. However, the index files corresponding to each sub-file have not yet been completely written into the root directory of the index file. At this point, the index files corresponding to each sub-file can be written into the root directory of the index file (overwriting or updating the index files corresponding to the split block files in the root directory of the block file) until all intermediate files of the index files in the temporary folder are written into their corresponding root directories, and then the temporary folder is cleared and the splitting environment is reset.

[0305] For example, in some specific embodiments, as shown in (e.g.) Figure 3 (As shown) The block file uses the xxx.fdb format structure as an example, such as Figure 14 As shown, when a merge error occurs during file splitting, the following steps are performed following the previously mentioned splitting process of xxx.fdb:

[0306] 1. If an error (procedural or logical, etc.) occurs during the segmentation process, terminate the segmentation immediately, report the error to the management personnel, and then proceed to step 10; otherwise, proceed to the next step.

[0307] 2. If a splitting error occurs due to a faulty blockchain node or a server crash followed by a restart, the file processing status will be retrieved from xxx.state (a marker file that records the file processing status). If the file processing status is splitting, proceed to step 3. If the file processing status is writing, proceed to step 7.

[0308] 3. Determine the file types present in the split folder (i.e., the temporary folder during the splitting process), and then perform adaptive processing based on the file types present in the split folder, as in any of steps 4 to 6.

[0309] 4. When both xxx.fdb (the sub-file) and xxx.json (the index file corresponding to the sub-file) exist in pairs in the split folder, it indicates that the splitting has started and has been going on for a period of time before stopping. Then, the following steps can be performed:

[0310] 4.1 First, determine if there is an error in parsing xxx.json;

[0311] 4.2 If a parsing error occurs, discard the incomplete index data in xxx.json and proceed to the next step; otherwise, if the parsing is successful, proceed directly to the next step.

[0312] 4.3 Verify the contents of xxx.fdb based on the information in xxx.json, and delete the data in xxx.fdb that does not have an index stored;

[0313] 4.4 The data structure has been repaired. Jump to step 3 in the aforementioned xxx.fdb splitting process to continue the previously unfinished splitting work;

[0314] 4.5 After completing the segmentation, proceed to step 10.

[0315] 5. When some xxx.fdb and xxx.json exist in pairs in the split folder, but there is only a single xxx.fdb, it indicates that the partial splitting has just begun and has ended. In this case, the single xxx.fdb can be deleted, and then the process can be skipped to step 3 in the aforementioned xxx.fdb splitting process to continue the unfinished splitting work until the splitting is completed, and then the process can be skipped to step 10.

[0316] 6. If there is only one xxx.fdb in the split folder, it means that the splitting has just started and has stopped. Then, jump to step 3 in the aforementioned xxx.fdb splitting process to continue the unfinished splitting work until the splitting is completed, and then jump to the subsequent step 10.

[0317] 7. Determine the file types present in the split folder (i.e., the temporary folder during the splitting process), and then perform adaptive processing based on the file types present in the split folder, as in any of steps 8 to 9.

[0318] 8. If both xxx.fdb and xxx.json exist in pairs in the split folder, it means that xxx.fdb has not yet been written into bfdb. Then, jump to step 7 in the aforementioned xxx.fdb splitting process to continue the unfinished splitting work until the splitting is completed, and then jump to the subsequent step 10.

[0319] 9. When some xxx.fdb and xxx.json exist in pairs in the split folder, but there is a single xxx.json, it indicates that xxx.json has not yet been completely written to KVDB. Then, jump to step 7 in the aforementioned xxx.fdb splitting process, write all the index information in split / 00000000000000000010.json that has not been written to KVDB to KVDB (e.g., leveldb), and continue the unfinished splitting work until the splitting is completed, then jump to the subsequent step 10.

[0320] 10. Clear the split file and reset the merge environment.

[0321] In summary, the file processing steps (including reading, writing, merging, and splitting) of this invention guarantee data integrity at every step of the execution process. Even if an abnormal interruption occurs at any step, restarting ensures that the data is not corrupted or lost. Furthermore, during merging and splitting, the operation is performed on the entire file as a unit, rather than at the granularity of individual blocks, minimizing operational costs and time. Specifically, file merging only involves merging files; it does not require separating multiple blocks of data stored within a file. This means that a file can contain multiple blocks of data, and the merging process of this invention, regardless of the number of blocks, simply merges the different files, achieving efficient archiving. The purpose of archiving is to improve service quality and ensure data manageability, thereby reducing the overall cost of operation, maintenance, and investment, maximizing storage utilization, and ensuring data accessibility.

[0322] like Figure 15 As shown, another aspect of this invention provides a block data processing apparatus 1500, which includes:

[0323] The first module 1510 is used to add a first check data segment and a block length field to each block data in the first original file storing block data to obtain a block file;

[0324] The second module 1520 is used to add a second check data segment and an index length field to each block index information in the second original file storing the block index information to obtain an index file;

[0325] The third module 1530 is used to respond to file processing requests for block files, and to process each block file according to the first checksum data segment, block length field and block data in the block file to obtain the file processing result; wherein, the file processing request includes file read request, file write request, file merge request and file split request.

[0326] The fourth module 1540 is used to update each index file based on the file processing results, combined with the second check data segment, index length field and index data in the index file.

[0327] Combined with appendix Figure 15 The specific implementation process of the block data processing device provided by the present invention is described as follows: First, the first module 1510 is used to add a first check data segment and a block length field to each block data in the first original file storing block data to obtain a block file; then, the second module 1520 is used to add a second check data segment and an index length field to each block index information in the second original file storing block index information to obtain an index file; next, the third module 1530 is used to respond to a file processing request for the block file, and perform corresponding processing on each block file according to the first check data segment, the block length field, and the block data in the block file to obtain a file processing result; specifically, the file processing request includes a file read request, a file write request, a file merge request, and a file split request; finally, the fourth module 1540 is used to update each index file according to the file processing result, combined with the second check data segment, the index length field, and the index data in the index file.

[0328] In some possible implementations, when the file processing request is a file merge request, the third module 1530 is further configured to: when a merge error occurs during the merging of block files, check the file types existing in the temporary folder: when the temporary folder contains a merge file, an index file, and a target file, repair the merge file, index file, and target file to complete the file merging of the block files; when the temporary folder contains only a merge file, delete the merge file to complete the file merging of the block files; when the temporary folder does not contain a merge file but contains an index file and a target file, write the data in the index file to the original folder, and delete the corresponding block files and index files from the temporary folder according to the file name of the target file to complete the file merging of the block files; when the temporary folder contains only a target file, delete the corresponding block files and index files from the temporary folder according to the file name of the target file to complete the file merging of the block files.

[0329] In some possible implementations, the apparatus may also include, but is not limited to, the following modules:

[0330] The fifth module is used to terminate the reading of the block file and discard all content after the erroneous data when the block data being read is erroneous.

[0331] The sixth module is used to terminate the writing process to the block file when a write error occurs while writing block data to the block file, and to read the end marker when restarting the writing process, and rewrite the block data starting from the last block data.

[0332] The seventh module is used to respond to file splitting requests for block files and record the file processing status of the block file as the file splitting status;

[0333] The eighth module is used to change the file processing status to file writing status when the file segmentation process of the block file is completed.

[0334] On the other hand, such as Figure 16 As shown, this embodiment of the invention also provides an electronic device 1600, which includes at least one processor 1610 and at least one memory 1620 for storing at least one program. The processor 1610 executes the program to perform the aforementioned block data processing method; taking one processor 1610 and one memory 1620 as an example.

[0335] The processor 1610 and memory 1620 can be connected via a bus or other means.

[0336] Memory 1620, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 1620 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1620 may optionally include memory remotely located relative to the processor, and this remote memory can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0337] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0338] Specifically, Figure 17 A schematic block diagram of a computer system architecture for implementing an electronic device according to embodiments of the present application is shown.

[0339] It should be noted that, Figure 17 The computer system 1700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0340] like Figure 17 As shown, the computer system 1700 includes a central processing unit (CPU) 1701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1702 or programs loaded from storage section 1708 into random access memory (RAM) 1703. The RAM 1703 also stores various programs and data required for system operation. The CPU 1701, ROM 1702, and RAM 1703 are interconnected via a bus 1704. An input / output interface 1705 (I / O interface) is also connected to the bus 1704.

[0341] The following components are connected to the input / output interface 1705: an input section 1706 including a keyboard, mouse, etc.; an output section 1707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a local area network card, modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the input / output interface 1705 as needed. Removable media 1711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on the drive 1710 as needed so that computer programs read from them can be installed into the storage section 1708 as needed.

[0342] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 2 The method shown.

[0343] In embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, as... Figure 17 The computer program can be downloaded and installed from a network via communication section 1709, and / or installed from removable media 1711. When the computer program is executed by central processing unit 1701, it performs the various functions defined in the system of this application.

[0344] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0345] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0346] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0347] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0348] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0349] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0350] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0351] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0352] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0353] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A block data processing method, applied to blockchain, characterized in that, include: Add a first checksum data segment and a block length field to each block of data in the first original file storing the block data to obtain the block file; A second check data segment and an index length field are added to each block index information in the second original file that stores the block index information to obtain the index file; In response to a file processing request for the block file, each block file is processed according to the first checksum segment, the block length field, and the block data in the block file to obtain a file processing result; wherein, the file processing request includes a file read request, a file write request, a file merge request, and a file split request; Based on the file processing results, each index file is updated according to the second verification data segment, index length field, and index data in the index file.

2. The block data processing method according to claim 1, characterized in that, The first verification data segment is used to verify the integrity of the block file; the block length field is used to characterize the field length of the block data in the block file; the second verification data segment is used to verify the integrity of the index file; the index length field is used to characterize the field length of the index information in the index file; The step of adding a first checksum data segment and a block length field to each block of data in the first original file storing the block data includes: Obtain the storage size of each block of data in the first original file containing the storage block data; For each block of data, a first verification data segment is added, and the block length of the block of data is calculated according to the corresponding storage size, and the block length field is added to the block of data.

3. The block data processing method according to claim 1, characterized in that, When the file processing request is a file read request, the response to the file processing request for the block file involves processing each block file accordingly based on the first checksum data segment, the block length field, and the block data in the block file, including: In response to a file read request for the block file, the file header of the block file is obtained, and the first checksum data segment and the block length field are read sequentially from the file header; Based on the block length field, data of the corresponding length is read from the block file to obtain block data; The verification information of the block data is calculated, and the verification information is compared with the first verification data segment to determine the integrity of the block data, until all block data in the block file is read.

4. The block data processing method according to claim 3, characterized in that, The method further includes: When reading block data from the block file results in erroneous data, the reading of the block file is terminated, and all content following the erroneous data is discarded.

5. The block data processing method according to claim 1, characterized in that, When the file processing request is a file write request, the response to the file processing request for the block file involves processing each block file accordingly based on the first checksum data segment, the block length field, and the block data in the block file, including: In response to a file write request for the block file, the first verification data segment, the block length field, and the block data are sequentially written into the block file; The index information corresponding to each block of data is written into the index file, and after all the block data is written, an end identifier is added to the index file. The end identifier is used to identify the last block of data written into the block file.

6. The block data processing method according to claim 5, characterized in that, The method further includes: When a write error occurs while writing block data to the block file, the write process to the block file is terminated, and when the write process is restarted, the end identifier is read, and the block data is rewritten starting from the last block data.

7. The block data processing method according to claim 1, characterized in that, When the file processing request is a file merge request, the response to the file processing request for the block file involves processing each block file accordingly based on the first checksum data segment, the block length field, and the block data in the block file, including: In response to a file merging request for the block file, several time-adjacent block files are obtained; The first block file among several adjacent block files is used as the merge file, and the other block files are merged into the merge file in sequence; wherein, each block data in the merge file includes the first check data segment and the block length field; Based on the information of the new block data in the merged file, the index file of the first block file is updated to obtain the index file corresponding to the merged file; Store the filenames of the individual block files to be merged into the merged file into the target file; Based on the filename of the target file, delete the corresponding block files and index files from the temporary folder; wherein, the temporary folder is used to store files during the processing.

8. The block data processing method according to claim 7, characterized in that, The step of using the first block file of a plurality of adjacent block files as the merge file, and sequentially merging the other block files into the merge file, includes: Based on the filenames of each block file, the block height of each block file is parsed, and based on the block height of each block file, each block file is determined to be a temporally adjacent block file; The first block file among the sequentially adjacent block files is used as the merge file, and the other block files are merged into the merge file in sequence.

9. A block data processing method according to claim 8, characterized in that, When the file processing request is a file merge request, the step of responding to the file processing request for the block file by performing corresponding processing on each block file according to the first checksum data segment, the block length field, and the block data in the block file further includes: When a merge error occurs while merging the block files, check the file types present in the temporary folder: When the merged file, the index file, and the target file all exist in the temporary folder, the merged file, the index file, and the target file are repaired to complete the file merging of the block file; When the temporary folder contains only the merged file, delete the merged file to complete the file merging of the block file; When the merged file does not exist in the temporary folder, but the index file and the target file exist, the data in the index file is written to the original folder, and according to the file name of the target file, each block file and the index file with the corresponding file name are deleted from the temporary folder to complete the file merging of the block files; When the target file exists only in the temporary folder, the corresponding block files and index files are deleted from the temporary folder according to the file name of the target file, thus completing the file merging of the block files.

10. A block data processing method according to claim 1, characterized in that, When the file processing request is a file splitting request, the response to the file processing request for the block file involves processing each block file accordingly based on the first checksum data segment, the block length field, and the block data in the block file, including: In response to a file segmentation request for the block file, determine the segmentation height and the number of segments in the block file; Based on the segmentation height and the number of segmented files, the block file is divided into a corresponding number of sub-files; wherein, each block data in the sub-file includes the first check data segment and the block length field; When a sub-file is obtained from each segmentation, an index file corresponding to the sub-file is configured.

11. A block data processing method according to claim 10, characterized in that, In response to a file segmentation request for the block file, determining the segmentation height and the number of segments of the block file includes at least one of the following: Identify the overall block height of the block file, and determine the height of each segment from the overall block height based on the number of segmented files parsed in the file segmentation request; or, Calculate the data size of the block file, and determine the number of files to be segmented and the segmentation height of the block file based on the size of the file to be segmented parsed in the file segmentation request.

12. The block data processing method according to claim 11, characterized in that, The method further includes: In response to a file segmentation request for the block file, the file processing status of the block file is recorded as a file segmentation status; When the file segmentation process of the block file is completed, the file processing status is changed to file writing status.

13. The block data processing method according to claim 12, characterized in that, When the file processing request is a file segmentation request, the step of responding to the file processing request for the block file by performing corresponding processing on each block file according to the first checksum data segment, the block length field, and the block data in the block file further includes: If a segmentation error occurs when splitting the block file, stop the file splitting operation and delete the generated sub-files and their corresponding index files. If a blockchain network failure occurs during the file splitting of the block file, data repair is performed based on the file processing status of the block file until the file splitting process of the block file is completed.

14. A block data processing method according to claim 13, characterized in that, When the file processing status of the block file is file splitting, and a blockchain network failure occurs during file splitting of the block file, data repair is performed based on the file processing status of the block file, including: When each of the sub-files obtained from the block file segmentation exists in pairs with the corresponding index file, the sub-files are repaired according to the index file, and the file segmentation process of the block file is restarted. When the block file is split into pairs of sub-files and index files, and a single sub-file is obtained, the single sub-file is deleted, and the file splitting process of the block file is restarted. When the block file is split into only one sub-file, the file splitting process for the block file is restarted directly.

15. A block data processing method according to claim 13, characterized in that, When the file processing status of the block file is in the file write state, and a blockchain network failure occurs during file splitting of the block file, data repair is performed based on the file processing status of the block file, including: When each of the sub-files obtained from the block file segmentation exists in pairs with its corresponding index file, the sub-files are transferred to the block management database, and the file segmentation process for the block file is restarted; wherein, the block management database is used to store the processed files; When the block file is split into pairs of sub-files and index files, and a single index file is obtained, the index file is written to the index database, and the file splitting process of the block file is restarted; wherein, the index database is used to store the index information of the block data.

16. A block data processing device, applied to blockchain, characterized in that, include: The first module is used to add a first check data segment and a block length field to each block data in the first original file storing block data to obtain a block file; The second module is used to add a second check data segment and an index length field to each block index information in the second original file storing the block index information, so as to obtain an index file; The third module is used to respond to file processing requests for the block files, and to perform corresponding processing on each block file according to the first checksum data segment, block length field and block data in the block files to obtain file processing results; wherein, the file processing requests include file read requests, file write requests, file merge requests and file split requests; The fourth module is used to update each of the index files based on the file processing results, the second verification data segment, the index length field, and the index data in the index files.

17. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 15.

Images