Big data processing method based on blockchain and related device

Abstract

This invention discloses a blockchain-based big data processing method and related apparatus. The method includes: constructing a data blockchain; the data blockchain includes at least one block corresponding to a storage device; each block includes at least one head node corresponding to a storage area; when a data block is stored in any storage device, querying its corresponding block and the head node corresponding to the storage area of ​​the data block, creating a new node corresponding to the data block under the head node, and storing the data digest and storage location of the data block in the node; updating the mapping table between nodes and data digests maintained by all blocks; when a data processing task for any data block is received, obtaining the data digest of the data block and querying the storage location of the data block using the data blockchain. This invention, by constructing a data blockchain to query the storage location of data blocks, uses the data blockchain as a distributed data storage index, achieving accurate and fast searching of big data.

Description

Technical Field

[0001] This invention relates to the field of big data technology, specifically to a blockchain-based big data processing method and apparatus, computing device and computer storage medium. Background Technology

[0002] With the advent of the mainframe era, centralized computer system architecture became mainstream. Due to the superior performance and stability of mainframes, their advantages in single-machine processing power were significant, leading IT systems to rapidly enter the centralized processing stage. The corresponding computer system is called a centralized system. A centralized system refers to a central node composed of one or more mainframe computers. Data is centrally stored on the central node, and all business units of the entire system are centrally deployed there. All system functions are centrally processed by this central node. In other words, in a centralized system, each terminal or client is only responsible for data input and output, while data storage and control processing are entirely handled by the mainframe. The biggest characteristic of a centralized system is its simple deployment structure. Because centralized systems are often based on high-performance mainframes, there is no need to consider how to deploy services across multiple nodes, and therefore no need to consider distributed collaboration issues between multiple nodes.

[0003] As computer systems become increasingly networked and miniaturized, traditional centralized systems are finding it increasingly difficult to meet the diverse needs of numerous users. Centralized systems are gradually being replaced by distributed systems. In distributed systems, computers can be deployed arbitrarily in space; for example, they may be installed in different racks, in different data centers, or even distributed across different cities. Because data storage in distributed systems is highly dispersed, the problem of accurately and quickly retrieving and processing stored data from these large and dispersed distributed resources urgently needs to be solved. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a blockchain-based big data processing method and apparatus, computing device and computer storage medium for solving the problem of how to accurately and quickly retrieve and process stored data from a large number of dispersed distributed resources.

[0005] According to one aspect of the present invention, a blockchain-based big data processing method is provided, comprising:

[0006] Construct a data blockchain; the data blockchain includes at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area;

[0007] When a data block is stored in any storage device, query the block corresponding to that storage device, and query the head node corresponding to the storage area of ​​the data block. Create a new node corresponding to the data block under the head node, and store the data digest and storage location of the data block in the node.

[0008] Update the mapping table between nodes and data digests maintained for all blocks;

[0009] When a data processing task is received for any data block, the data digest of the data block is obtained, and the storage location of the data block is queried using the data blockchain.

[0010] In an alternative approach, the method further includes:

[0011] If no block corresponding to the storage device is found, a new block corresponding to the storage device will be created.

[0012] If no header node corresponding to the storage area of ​​the data block is found, a new header node corresponding to the storage area of ​​the data block is created under the block corresponding to the storage device.

[0013] In one alternative approach, the data blockchain further includes: a management block; the step of obtaining a data digest of the data block and querying the storage location of the data block using the data blockchain when a data processing task for any data block is received further includes:

[0014] After obtaining the data digest of the data block, the management block broadcasts the data digest to each block.

[0015] After receiving the data digest, each block determines whether there is a matching data digest in the mapping table it maintains, and sends the query result back to the management block.

[0016] The management block determines the storage location of the data block based on the query results returned by each block.

[0017] In an alternative approach, after broadcasting the data digest to each block, the method further includes:

[0018] Management block start timer;

[0019] The management block retrieves the query results returned by each block within the time limit.

[0020] In an optional approach, the management block determining the storage location of the data block based on the query results returned by each block further includes:

[0021] The management block determines whether the number of query results matches the total number of blocks;

[0022] If there is a discrepancy, the management block will broadcast the data digest to each block again and receive the query results from each block. It will then determine whether the number of received query results is consistent with the total number of blocks. If there is a discrepancy, it will send a message that the data block is incorrect.

[0023] If the number of query results is the same as the total number of blocks, determine whether all query results are the same. If they are the same, determine and return the storage location of the data block.

[0024] In one optional approach, the step of determining whether all query results are the same if the number of query results matches the total number of blocks, and then determining and reporting the storage location of the data block if they are the same, further includes:

[0025] If it is determined that all query results are not completely identical, the management block broadcasts the data digest to each block again and receives the query results from each block.

[0026] Determine the number of each type of query result in the two query results respectively, and group the same query results into one category;

[0027] If the two query results are the same and the number of each type of query result is consistent, then the type of query result with the largest number is determined as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0028] If the two query results are the same but the number of each type of query result is different, then the type of query result with the largest number of results is selected as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0029] If the two query results are different, the query result with the fewest categories is determined, and the query result with the most categories is selected as the target category query result. The storage location of the data block is determined and fed back based on the target category query result.

[0030] In an optional approach, determining and returning the storage location of the data block based on the target class query result further includes:

[0031] Calculate the ratio of the number of results for the target class to the total number of results.

[0032] Calculate the comparison value based on the number of categories in the query results;

[0033] If the ratio is greater than or equal to the comparison value, then the storage location corresponding to the target class query result is determined to be the storage location of the data block;

[0034] If the ratio is less than the comparison value, an error message is sent to the feedback data block.

[0035] According to another aspect of the present invention, a blockchain-based big data processing apparatus is provided, comprising:

[0036] A construction module is used to construct a data blockchain; the data blockchain includes at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area; when a data block is stored in any storage device, the module queries the block corresponding to the storage device and queries the head node corresponding to the storage area of ​​the data block, creates a new node corresponding to the data block under the head node, and stores the data digest and storage location of the data block in the node;

[0037] The update module is used to update the mapping table between nodes and data digests maintained by all blocks;

[0038] The query module is used to obtain the data digest of any data block when a data processing task is received, and to query the storage location of the data block using the data blockchain.

[0039] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0040] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described blockchain-based big data processing method.

[0041] According to another aspect of the present invention, a computer storage medium is provided, wherein at least one executable instruction is stored therein, the executable instruction causing a processor to perform operations corresponding to the above-described blockchain-based big data processing method.

[0042] The solution provided in the above embodiments of the present invention constructs a data blockchain. The data blockchain includes at least one block corresponding to a storage device. Each block includes at least one head node corresponding to a storage area. When a data block is stored in any storage device, the block corresponding to that storage device is queried, and the head node corresponding to the storage area of ​​the data block is queried. A new node corresponding to the data block is created under the head node, and the data digest and storage location of the data block are stored in the node. The mapping table between nodes and data digests maintained by all blocks is updated. When a data processing task for any data block is received, the data digest of the data block is obtained, and the storage location of the data block is queried using the data blockchain. Based on the existing blockchain structure, the present invention improves and optimizes the data structure and data storage of the data blockchain. By constructing a data blockchain to query the storage location of data blocks, the data blockchain is used as a data storage index for distributed storage, realizing accurate and fast searching of big data.

[0043] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0045] Figure 1 A flowchart illustrating a blockchain-based big data processing method according to an embodiment of the present invention is shown.

[0046] Figure 2 A flowchart illustrating another embodiment of the blockchain-based big data processing method of the present invention is shown.

[0047] Figure 3 A schematic diagram of the data blockchain structure according to an embodiment of the present invention is shown;

[0048] Figure 4 A schematic diagram of the structure of a blockchain-based big data processing device according to an embodiment of the present invention is shown;

[0049] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown. Detailed Implementation

[0050] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0051] Figure 1 This diagram illustrates a flowchart of a blockchain-based big data processing method according to an embodiment of the present invention. The method constructs a data blockchain consisting of a management block and multiple blocks, where each block corresponds to a storage device. When data retrieval and processing are required, the storage location of the data is queried from each block, and the final storage location is determined based on the query results returned by each block. Specifically, as shown... Figure 1 As shown, it includes the following steps:

[0052] Step S101: Construct a data blockchain; the data blockchain includes at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area.

[0053] Most blockchains currently employ a chain-like structure based on blocks in their data structure design. Each block in a blockchain consists of a block header and a block body. The block header stores data such as the Merkle root, previous block hash, and timestamp, while the block body stores batch transaction data. In terms of data storage design, blockchain index data and state data are typically stored in key-value databases. For example, Bitcoin and Ethereum use LevelDB (a lightweight, single-machine database) to store index data. Blockchain is a decentralized, immutable, distributed database with high write performance but lower retrieval efficiency, especially for non-key (non-prime, such as address, digest description, etc.) queries.

[0054] In this embodiment, the data structure and data storage of the data blockchain are improved and optimized, and the data blockchain is used as a data storage index for distributed storage to achieve accurate and fast searching of big data.

[0055] Specifically, a data blockchain is constructed, comprising at least one block corresponding to a data storage device; each block includes at least one head node corresponding to a storage area. Optionally, the data blockchain also includes a management block, which is connected to the data storage device and used to record all data stored on the data storage device. Optionally, the management block is connected to a data processing device and used to query the location of data to be processed.

[0056] like Figure 3As shown, the data blockchain includes multiple blocks (such as block 1, block 2, etc.), and each block (such as block 1) corresponds to a data storage device (such as storage device 1). When a new data storage device is added, a new block is created in the data blockchain to correspond to it through the management block. Each block includes at least one head node corresponding to a storage area (such as block 2 including head node 21 corresponding to storage area 21 and head node 22 corresponding to storage area 22).

[0057] Furthermore, if a data storage device stores data, a head node (such as head node 11, head node 21, etc.) is created under the corresponding block of that data storage device. This head node corresponds to a storage area of ​​the data storage device (such as storage area 21, storage area 22). That is, if the data storage device includes multiple storage areas, multiple head nodes are connected under the corresponding block of the data storage device, with each head node corresponding to one storage area. If a data storage device has only one storage area, only one head node is connected under the corresponding block of the data storage device. If the data storage device does not store data, only blocks are stored, and no head node exists.

[0058] Furthermore, the head node (e.g., head node 21) connects to nodes (e.g., node 211), each node corresponding to a data block or data table stored in its corresponding data storage area. The specific method for determining the data block can be determined according to the data storage user's needs. Each node stores a data digest (e.g., data digest 211) and a storage location (e.g., storage location 211) for that data block. The data digest describes the data content, and the storage location determines the location of the data block. When a data block is stored, the corresponding data storage device informs the user of the data block's data digest and storage location through a management block.

[0059] Step S102: When a data block is stored in any storage device, query the block corresponding to the storage device and query the head node corresponding to the storage area of ​​the data block. Create a new node corresponding to the data block under the head node and store the data digest and storage location of the data block in the node.

[0060] When a data block is stored in any data storage device, the management block query checks whether a block corresponding to that data storage device exists, and queries the head node corresponding to the storage area of ​​the data block. A new node corresponding to the data block is created under the head node, and the data digest and storage location of the data block are stored in the node.

[0061] Step S103: Update the mapping table between nodes and data digests maintained by all blocks.

[0062] To quickly find data, managing blocks and / or all blocks requires maintaining the correspondence between corresponding nodes and data digests. For example, after fingerprinting the nodes and data digests, the correspondence between them is established through a mapping table (such as a hash table).

[0063] Step S104: When a data processing task for any data block is received, obtain the data digest of the data block and use the data blockchain to query the storage location of the data block.

[0064] When a data block processing task (such as data retrieval) is obtained, the data digest of the data block is obtained, the storage location of the data block is queried using the data blockchain, and the storage location of the data blocks in each blockchain is fed back through the management block.

[0065] The solution provided in the above embodiments of the present invention constructs a data blockchain. The data blockchain includes at least one block corresponding to a storage device. Each block includes at least one head node corresponding to a storage area. When a data block is stored in any storage device, the block corresponding to that storage device is queried, and the head node corresponding to the storage area of ​​the data block is queried. A new node corresponding to the data block is created under the head node, and the data digest and storage location of the data block are stored in the node. The mapping table between nodes and data digests maintained by all blocks is updated. When a data processing task for any data block is received, the data digest of the data block is obtained, and the storage location of the data block is queried using the data blockchain. Based on the existing blockchain structure, the present invention improves and optimizes the data structure and data storage of the data blockchain. By constructing a data blockchain to query the storage location of data blocks, the data blockchain is used as a data storage index for distributed storage, achieving accurate and fast searching of big data.

[0066] Figure 2 A flowchart illustrating another embodiment of the blockchain-based big data processing method of the present invention is shown. This method determines the storage location of a data block based on the number of query results received by the management block and the total number of blocks. Furthermore, it determines and feeds back the storage location of the data block based on the number of query results received twice by the management block and the number of various types of query results. Specifically, as... Figure 2 As shown, it includes the following steps:

[0067] Step S201: Construct a data blockchain; the data blockchain includes at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area.

[0068] Alternatively, the individual blocks in the data blockchain can be maintained by managing blocks (e.g., by creating, deleting, modifying, merging, or splitting blocks).

[0069] Optionally, each head node can be configured with two pointers, one pointing to the direct successor node and the other to the direct predecessor node. For example... Figure 3 As shown, the head node (e.g., head node 21) connects multiple nodes through a linked structure (e.g., node 212 is the successor of node 211). This linked structure is more suitable for traversing from front to back, but not from back to front. By setting two pointers for each node, it is easy to access its predecessor and successor nodes starting from any head node or any other node.

[0070] Step S202: When a data block is stored in any storage device, query the block corresponding to the storage device and query the head node corresponding to the storage area of ​​the data block. Create a new node corresponding to the data block under the head node and store the data digest and storage location of the data block in the node.

[0071] In one alternative approach, if no block corresponding to the storage device is found, a new block corresponding to the storage device is created; if a block is found, the block is located.

[0072] If no header node corresponding to the storage area of ​​the data block is found, a new header node corresponding to the storage area of ​​the data block is created under the block corresponding to the storage device. If a header node is found, the header node is located.

[0073] Step S203: Update the mapping table between nodes and data digests maintained by all blocks.

[0074] The mapping table between nodes and data digests is updated and maintained through all blocks and / or the management block. Optionally, the mapping table between the management block and all blocks is periodically checked to ensure data consistency.

[0075] In step S204, after the management block obtains the data digest of the data block, it broadcasts the data digest to each block. After receiving the data digest, each block determines whether there is a matching data digest in its own mapping table and returns the query result to the management block.

[0076] When a data processing task is received for any data block, the management block broadcasts a data digest of the data block to all other blocks. Each block determines whether a matching data digest exists based on its maintained mapping table and reports the query result back to the management block. For example, after obtaining the description information of the data block (such as a data digest or data fingerprint), the management block broadcasts this description information to all other blocks. After receiving this description information, all blocks determine whether a data digest corresponding to the data block description information exists based on their maintained mapping table (such as a correspondence table) and report the query result back to the management block.

[0077] Furthermore, if each block finds a data digest corresponding to the data block description information, it reports the associated information, including the block, head node, node, and data digest, to the management block. If no data digest corresponding to the data block description information is found, it reports "no result found" to the management block. For example, ... Figure 3 If block 2 is found to contain a data digest 211 corresponding to the data block description information, the associated information corresponding to the data digest 211 is fed back to the management block as follows: Block 2 - Head node 21 - Node 211 - Data digest 211 - Storage location 211.

[0078] In one alternative approach, after broadcasting the data digest to each block, the method further includes:

[0079] Management block start timer;

[0080] The management block retrieves the query results returned by each block within the time limit.

[0081] For example, the management block starts a countdown timer after or simultaneously with broadcasting the data block description information. The countdown timer has a duration of t, and the query results from each block are retrieved within that time. To set the timer reasonably, the formula for setting the time t is as follows, based on the amount of data in the data block description information, the total number of blocks, and the maximum number of nodes connected to a single head node:

[0082] t = (Data volume of block description information / α) × Total number of blocks × Maximum number of nodes connected to a single head node × β, where α represents data capacity in GB and β represents time in milliseconds.

[0083] For example, based on experience, if parameters α = 1 and β = 20 are set, then the formula for time t is:

[0084] t = (Data volume of block description information / 1G) × Total number of blocks × Maximum number of nodes connected to a single head node × 20 milliseconds.

[0085] Once the countdown timer ends, the management block stops receiving information from the block feedback and determines the number of query results received.

[0086] Step S205: The management block determines whether the number of query results is consistent with the total number of blocks.

[0087] The storage location of the database is determined based on the number of query results received from each block in the management block and the total number of blocks.

[0088] Specifically, the management block determines whether the number of query results is consistent with the total number of blocks. If they are inconsistent, it indicates that there is an error in the confirmation process, and step S206 is executed; if they are consistent, step S208 is executed.

[0089] In step S206, the management block broadcasts the data digest to each block again, receives the query results from each block, and then determines whether the number of received query results is consistent with the total number of blocks.

[0090] The management block receives the query results from each block again. Specifically, the management block broadcasts the data digest to each block again and receives the query results from each block. It then determines whether the number of received query results is consistent with the total number of blocks. If they are consistent, step S208 is executed; otherwise, step S207 is executed, which sends a message that there is an error in the data block.

[0091] Step S207: Feedback message indicating an error in the data block.

[0092] If a data block is found to be incorrect, an error alert will be issued to confirm whether the data block has been stored correctly.

[0093] Step S208: Determine whether all query results are the same. If they are the same, determine and return the storage location of the data block.

[0094] Determine if all query results are the same. If they are the same, determine and return the storage location of the data block, and read the data block information based on the storage location.

[0095] In one alternative approach, if it is determined that all query results are not completely identical, the management block broadcasts the data digest to each block again and receives the query results from each block.

[0096] Determine the number of each type of query result in the two query results respectively, and group the same query results into one category;

[0097] If the two query results are the same and the number of each type of query result is consistent, then the type of query result with the largest number is determined as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0098] If the two query results are the same but the number of each type of query result is different, then the type of query result with the largest number of results is selected as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0099] If the two query results are different, the query result with the fewest number of result categories is determined, and the query result with the largest number of result categories is selected as the target query result. The storage location of the data block is determined and reported based on the target query result.

[0100] For example, the first query result is: there are a total of N query results, of which N-1 query results result in node 221, and only 1 query result results in node 211. Then, the different types of query results and the number of each type of query results are determined as follows: one type of query result is node 221, and its quantity is N-1, and the other type of query result is node 211, and its quantity is 1. The number of query result types is 2.

[0101] Since the results of all queries in the first round are not completely identical, the management block broadcasts the data digest to each block again and receives the query results from each block to obtain the second round of different query results and the number of results for each type.

[0102] If the second query result is exactly the same as the first, and there are a total of N query results, of which N-1 query results result in node 221 and only 1 result results in node 211, that is, the query results are the same and the number of each type of query result is the same, then select the type of query result with the largest number, that is, select N-1 query results as node 221, as the target type of query result.

[0103] If the second query result is: there are a total of N query results, of which N-2 query results result in node 221 and 2 query results result in node 211, that is, the query results are the same and the number of each type of query result is different, then select the type of query result with the largest number from the two query results, that is, select N-1 query results as node 221, as the target type of query result.

[0104] If the second query result is: there are a total of N query results, of which N-2 query results are node 221, 1 query result is node 211, and 1 query result is 212, and the number of categories of the query results is more than that of the first query, then the query result with the fewest categories is determined, that is, the first query result. From the first query result, the query result with the largest number of categories is selected, that is, N-1 query results are selected as node 221, as the target category query result.

[0105] In one alternative approach, determining and returning the storage location of the data block based on the target class query results further includes:

[0106] Calculate the ratio of the number of results for the target class to the total number of results.

[0107] Calculate the comparison value based on the number of categories in the query results;

[0108] If the ratio is greater than or equal to the comparison value, then the storage location corresponding to the target class query result is determined as the storage location of the data block;

[0109] If the ratio is less than the comparison value, an error message is sent to the data block.

[0110] For example, if two queries yield the same results and the number of results for each category is also the same, the ratio of the number of the most frequent query result (i.e., the target category query result) to the total number of query results is calculated. If this ratio is greater than or equal to (1 - number of query result categories / N), the storage location of the node corresponding to the most frequent query result is returned; otherwise, an error message is returned regarding the data block. In the example above, the ratio of the number of target category query results to the total number of query results is (N-1) / N, and the comparison value is 1-2 / N. Since the former is greater than the latter, storage location 221 is determined as the storage location of the data block.

[0111] If two query results are identical but the number of results for each category differs, determine the maximum number of results for the same category (i.e., the target category query results). Calculate the ratio of the number of target category query results to the total number of query results. If this ratio is greater than or equal to (1 - number of query result categories / N), then the storage location of the node corresponding to the target category query result is returned; otherwise, an error message is returned regarding the data block. In the example above, the ratio of the number of target category query results to the total number of query results is (N-1) / N, and the comparison value is 1-2 / N. Since the former is greater than the latter, storage location 221 is determined as the storage location of the data block.

[0112] If the two query results are different, determine the result with the fewest query result classes. Calculate the ratio of the number of query results with the most occurrences in that result (i.e., the target class query result) to the total number of query results. If this ratio is greater than or equal to (1 - number of query result classes / N), where the number of query result classes is the number of query result classes in the result with the fewest occurrences, then the storage location of the node corresponding to the target class query result is returned; otherwise, an error message is returned indicating a data block problem. In the example above, the ratio of the number of target class query results to the total number of query results is (N-1) / N, and the comparison value is 1-2 / N. Since the former is greater than the latter, storage location 221 is determined as the storage location of the data block.

[0113] The solution provided in the above embodiments of the present invention improves the accuracy of data block queries by determining the number of query results and the total number of blocks through a management block, and by determining the storage location of data blocks based on the two sets of query results and the number of each type of query result. The management block determines the number of received query results based on the time it takes for each block to return query results, and combines this with the ratio of the number of target type query results to the total number of query results to determine the storage location of the data blocks. This improves data block query efficiency while maintaining accuracy.

[0114] Figure 4A schematic diagram of the structure of a blockchain-based big data processing device according to an embodiment of the present invention is shown. The blockchain-based big data processing device 400 includes: a construction module 410, an update module 420, and a query module 430.

[0115] The construction module 410 is used to construct a data blockchain; the data blockchain includes at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area; when a data block is stored in any storage device, the block corresponding to the storage device is queried, and the head node corresponding to the storage area of ​​the data block is queried. A new node corresponding to the data block is created under the head node, and the data digest and storage location of the data block are stored in the node.

[0116] The update module 420 is used to update the mapping table between nodes and data digests maintained by all blocks;

[0117] The query module 430 is used to obtain the data digest of any data block when a data processing task for any data block is received, and to query the storage location of the data block using the data blockchain.

[0118] In an alternative embodiment, the building module 410 is further configured to:

[0119] If no block corresponding to the storage device is found, a new block corresponding to the storage device will be created.

[0120] If no header node corresponding to the storage area of ​​the data block is found, a new header node corresponding to the storage area of ​​the data block is created under the block corresponding to the storage device.

[0121] In an alternative embodiment, the query module 430 is further configured to:

[0122] After obtaining the data digest of the data block, the management block broadcasts the data digest to each block;

[0123] After receiving the data digest, each block determines whether there is a matching data digest in the mapping table it maintains, and sends the query result back to the management block.

[0124] The management block determines the storage location of the data block based on the query results returned by each block.

[0125] In an alternative embodiment, the query module 430 is further configured to:

[0126] Management block start timer;

[0127] The management block retrieves the query results returned by each block within the time limit.

[0128] In an alternative embodiment, the query module 430 is further configured to:

[0129] The management block determines whether the number of query results matches the total number of blocks;

[0130] If there is a discrepancy, the management block will broadcast the data digest to each block again and receive the query results from each block. It will then determine whether the number of received query results is consistent with the total number of blocks. If there is a discrepancy, it will send a message that the data block is incorrect.

[0131] If the number of query results is the same as the total number of blocks, determine whether all query results are the same. If they are the same, determine and return the storage location of the data block.

[0132] In an alternative embodiment, the query module 430 is further configured to:

[0133] If it is determined that all query results are not completely identical, the management block broadcasts the data digest to each block again and receives the query results from each block.

[0134] Determine the number of each type of query result in the two query results respectively, and group the same query results into one category;

[0135] If the two query results are the same and the number of each type of query result is consistent, then the type of query result with the largest number is determined as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0136] If the two query results are the same but the number of each type of query result is different, then the type of query result with the largest number of results is selected as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0137] If the two query results are different, the query result with the fewest categories is determined, and the query result with the most categories is selected as the target category query result. The storage location of the data block is determined and fed back based on the target category query result.

[0138] In an alternative embodiment, the query module 430 is further configured to:

[0139] Calculate the ratio of the number of results for the target class to the total number of results.

[0140] Calculate the comparison value based on the number of categories in the query results;

[0141] If the ratio is greater than or equal to the comparison value, then the storage location corresponding to the target class query result is determined to be the storage location of the data block;

[0142] If the ratio is less than the comparison value, an error message is sent to the feedback data block.

[0143] The solution provided in the above embodiments of the present invention constructs a data blockchain. The data blockchain includes at least one block corresponding to a storage device. Each block includes at least one head node corresponding to a storage area. When a data block is stored in any storage device, the block corresponding to that storage device is queried, and the head node corresponding to the storage area of ​​the data block is queried. A new node corresponding to the data block is created under the head node, and the data digest and storage location of the data block are stored in the node. The mapping table between nodes and data digests maintained by all blocks is updated. When a data processing task for any data block is received, the data digest of the data block is obtained, and the storage location of the data block is queried using the data blockchain. Based on the existing blockchain structure, the present invention improves and optimizes the data structure and data storage of the data blockchain. By constructing a data blockchain to query the storage location of data blocks, the data blockchain is used as a data storage index for distributed storage, achieving accurate and fast searching of big data.

[0144] Figure 5 The diagram shows a structural schematic of an embodiment of the computing device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0145] like Figure 5 As shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0146] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508. Communication interface 504 is used to communicate with other network elements such as clients or other servers. The processor 502 executes program 510, specifically performing the relevant steps in the above-described embodiment of the blockchain-based big data processing method.

[0147] Specifically, program 510 may include program code that includes computer operation instructions.

[0148] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0149] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0150] Specifically, program 510 can be used to cause processor 502 to perform the following operations:

[0151] Construct a data blockchain; the data blockchain includes at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area;

[0152] When a data block is stored in any storage device, query the block corresponding to that storage device, and query the head node corresponding to the storage area of ​​the data block. Create a new node corresponding to the data block under the head node, and store the data digest and storage location of the data block in the node.

[0153] Update the mapping table between nodes and data digests maintained for all blocks;

[0154] When a data processing task is received for any data block, the data digest of the data block is obtained, and the storage location of the data block is queried using the data blockchain.

[0155] In an alternative manner, the program 510 causes the processor to perform the following operations:

[0156] If no block corresponding to the storage device is found, a new block corresponding to the storage device will be created.

[0157] If no header node corresponding to the storage area of ​​the data block is found, a new header node corresponding to the storage area of ​​the data block is created under the block corresponding to the storage device.

[0158] In an alternative manner, the program 510 causes the processor to perform the following operations:

[0159] After obtaining the data digest of the data block, the management block broadcasts the data digest to each block;

[0160] After receiving the data digest, each block determines whether there is a matching data digest in the mapping table it maintains, and sends the query result back to the management block.

[0161] The management block determines the storage location of the data block based on the query results returned by each block.

[0162] In an alternative manner, after broadcasting the data digest to each block, the program 510 causes the processor to perform the following operations:

[0163] Management block start timer;

[0164] The management block retrieves the query results returned by each block within the time limit.

[0165] In an alternative manner, the program 510 causes the processor to perform the following operations:

[0166] The management block determines whether the number of query results matches the total number of blocks;

[0167] If there is a discrepancy, the management block will broadcast the data digest to each block again and receive the query results from each block. It will then determine whether the number of received query results is consistent with the total number of blocks. If there is a discrepancy, it will send a message that the data block is incorrect.

[0168] If the number of query results is the same as the total number of blocks, determine whether all query results are the same. If they are the same, determine and return the storage location of the data block.

[0169] In an alternative manner, the program 510 causes the processor to perform the following operations:

[0170] If it is determined that all query results are not completely identical, the management block broadcasts the data digest to each block again and receives the query results from each block.

[0171] Determine the number of each type of query result in the two query results respectively, and group the same query results into one category;

[0172] If the two query results are the same and the number of each type of query result is consistent, then the type of query result with the largest number is determined as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0173] If the two query results are the same but the number of each type of query result is different, then the type of query result with the largest number of results is selected as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

[0174] If the two query results are different, the query result with the fewest categories is determined, and the query result with the most categories is selected as the target category query result. The storage location of the data block is determined and fed back based on the target category query result.

[0175] In an alternative manner, the program 510 causes the processor to perform the following operations:

[0176] Calculate the ratio of the number of results for the target class to the total number of results.

[0177] Calculate the comparison value based on the number of categories in the query results;

[0178] If the ratio is greater than or equal to the comparison value, then the storage location corresponding to the target class query result is determined to be the storage location of the data block;

[0179] If the ratio is less than the comparison value, an error message is sent to the feedback data block.

[0180] The solution provided in the above embodiments of the present invention constructs a data blockchain. The data blockchain includes at least one block corresponding to a storage device. Each block includes at least one head node corresponding to a storage area. When a data block is stored in any storage device, the block corresponding to that storage device is queried, and the head node corresponding to the storage area of ​​the data block is queried. A new node corresponding to the data block is created under the head node, and the data digest and storage location of the data block are stored in the node. The mapping table between nodes and data digests maintained by all blocks is updated. When a data processing task for any data block is received, the data digest of the data block is obtained, and the storage location of the data block is queried using the data blockchain. Based on the existing blockchain structure, the present invention improves and optimizes the data structure and data storage of the data blockchain. By constructing a data blockchain to query the storage location of data blocks, the data blockchain is used as a data storage index for distributed storage, achieving accurate and fast searching of big data.

[0181] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute the blockchain-based big data processing method in any of the above method embodiments.

[0182] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0183] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0184] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0185] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0186] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0187] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0188] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A blockchain-based big data processing method, characterized in that, The method includes: Construct a data blockchain; the data blockchain includes management blocks and at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area; When a data block is stored in any storage device, query the block corresponding to that storage device, and query the head node corresponding to the storage area of ​​the data block. Create a new node corresponding to the data block under the head node, and store the data digest and storage location of the data block in the node. Update the mapping table between nodes and data digests maintained for all blocks; When a data processing task for any data block is received, a data digest for that data block is obtained, and the storage location of the data block is queried using the data blockchain. Specifically, after obtaining the data digest, the management block broadcasts the data digest to all other blocks. Each block, upon receiving the data digest, determines whether a matching data digest exists in its maintained mapping table and reports the query result back to the management block. The management block checks if the number of query results matches the total number of blocks. If not, the management block broadcasts the data digest to all other blocks again and receives the query results from each block, again checking if the number of received query results matches the total number of blocks. If not, it reports an error message indicating a data block is not present. If the number of query results matches the total number of blocks, it checks if all query results are identical. If identical, it determines and reports the storage location of the data block. If all query results are not identical... Similarly, the management block broadcasts the data digest to each block again and receives the query results from each block; it determines the number of each type of query result in the two query results, and groups the same query results into one category; if the two query results are the same and the number of each type of query result is consistent, the type of query result with the largest number is determined as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result; if the two query results are the same but the number of each type of query result is inconsistent, the type of query result with the largest number is selected from the two query results as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result; if the two query results are different, the query result with the fewest number of query result categories is determined, the type of query result with the largest number is selected from that query result as the target type query result, and the storage location of the data block is determined and fed back based on the target type query result.

2. The method according to claim 1, characterized in that, The method further includes: If no block corresponding to the storage device is found, a new block corresponding to the storage device will be created. If no header node corresponding to the storage area of ​​the data block is found, a new header node corresponding to the storage area of ​​the data block is created under the block corresponding to the storage device.

3. The method according to claim 1, characterized in that, After broadcasting the data digest to each block, the method further includes: Management block start timer; The management block retrieves the query results returned by each block within the time limit.

4. The method according to claim 1, characterized in that, The step of determining and returning the storage location of the data block based on the target class query results further includes: Calculate the ratio of the number of results for the target class to the total number of results. Calculate the comparison value based on the number of categories in the query results; If the ratio is greater than or equal to the comparison value, then the storage location corresponding to the target class query result is determined to be the storage location of the data block; If the ratio is less than the comparison value, an error message is sent to the feedback data block.

5. A blockchain-based big data processing device, characterized in that, The device includes: A construction module is used to construct a data blockchain; the data blockchain includes management blocks and at least one block corresponding to a storage device; the block includes at least one head node corresponding to a storage area; when a data block is stored in any storage device, the module queries the block corresponding to the storage device and queries the head node corresponding to the storage area of ​​the data block, creates a new node corresponding to the data block under the head node, and stores the data digest and storage location of the data block in the node; The update module is used to update the mapping table between nodes and data digests maintained by all blocks; The query module is used to obtain a data digest of any data block when a data processing task is received, and to query the storage location of the data block using the data blockchain. Specifically, after obtaining the data digest, the management block broadcasts the data digest to all other blocks. Each block, upon receiving the data digest, determines whether a matching data digest exists in its maintained mapping table and returns the query result to the management block. The management block checks if the number of query results matches the total number of blocks. If not, the management block broadcasts the data digest to all other blocks again and receives the query results from each block, again checking if the number of received query results matches the total number of blocks. If not, it reports an error message indicating a data block is not present. If the number of query results matches the total number of blocks, it checks if all query results are identical. If identical, it determines and returns the storage location of the data block. If all query results are identical... If the results are not entirely the same, the management block broadcasts the data digest to all blocks again and receives the query results from each block. It then determines the number of each type of query result in the two sets of results, grouping identical results into one category. If the two sets of results are identical and the number of each type is the same, the type with the most results is designated as the target type, and the storage location of the data block is determined and reported based on this target type. If the two sets of results are identical but the number of each type is different, the type with the most results is selected as the target type, and the storage location of the data block is determined and reported based on this target type. If the two sets of results are different, the result with the fewest types is selected, and the type with the most results from this result is selected as the target type, and the storage location of the data block is determined and reported based on this target type.

6. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the blockchain-based big data processing method as described in any one of claims 1-4.

7. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the blockchain-based big data processing method as described in any one of claims 1-4.

Images