Data processing method and device based on distributed ledger, electronic device and storage medium

Abstract

The present disclosure provides a distributed ledger-based data processing method and device, electronic equipment and storage medium, relating to the technical field of computer, which comprises the following steps: receiving a computing task sent by a master computing node in a distributed computing cluster; determining a blockchain node associated with a slave computing node and loading ledger data on the blockchain node; performing computation on the ledger data to generate a first computing result; and sending the first computing result to the master computing node for aggregation to obtain a second computing result. Through the binding of the slave computing node and the blockchain node, single computation, and aggregation by the master computing node, the overall analysis of the ledger data on the blockchain node can be realized, and the quality and use value of the data processing result can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device and storage medium based on distributed ledger. Background Technology

[0002] In essence, blockchain is not a completely new invention, but rather an innovative integration of various existing technologies, including cryptography, distributed storage, consensus mechanisms, and P2P networks. Through the combination of these technologies, blockchain possesses characteristics such as traceability, tamper-proofness, and security and trustworthiness, cleverly solving the problem of trusted collaboration among multiple parties. Therefore, blockchain is currently being widely applied in fields such as finance, government affairs, and supply chains. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, one objective of this disclosure is to propose a data processing method based on distributed ledgers.

[0005] The second objective of this disclosure is to propose a data processing apparatus based on a distributed ledger.

[0006] The third objective of this disclosure is to propose an electronic device.

[0007] The fourth objective of this disclosure is to provide a non-transitory computer-readable storage medium.

[0008] The fifth objective of this disclosure is to provide a computer program product.

[0009] To achieve the above objectives, the first aspect of this disclosure proposes a data processing method based on a distributed ledger, executed by a slave computing node in a distributed computing cluster, comprising: receiving a computing task sent by a master computing node in the distributed computing cluster; determining the blockchain node associated with the slave computing node and loading ledger data on the blockchain node; performing calculations on the ledger data to generate a first calculation result; and sending the first calculation result to the master computing node for aggregation to obtain a second calculation result.

[0010] According to one embodiment of this disclosure, determining a blockchain node associated with a computing node includes: determining, from a host cluster, the hosts deployed on the computing node; and determining, from the blockchain cluster, a blockchain node associated with the computing node based on the hosts deployed on the computing node, wherein each host in the host cluster has a blockchain node deployed in the blockchain cluster.

[0011] According to one embodiment of this disclosure, before loading ledger data on a blockchain node, the method further includes: calling a format conversion program to obtain ledger data in a first format stored on the blockchain node; and deserializing and converting the ledger data in the first format to obtain ledger data in a second format that can be loaded from the computing node.

[0012] According to one embodiment of this disclosure, a format conversion program obtains ledger data in a first format stored locally by a blockchain node, including: obtaining the storage address of the ledger data in the first format, and reading the ledger data in the first format from the blockchain node according to the storage address.

[0013] According to one embodiment of this disclosure, calculations are performed on ledger data to generate a first calculation result, including: obtaining transaction data for each transaction based on the ledger data; extracting metadata from the transaction data according to the transaction data structure tree to obtain transaction data to be analyzed; and executing pre-written calculation logic code on the transaction data to be analyzed for all transactions on the blockchain node to obtain the first calculation result.

[0014] According to one embodiment of this disclosure, metadata extraction is performed on transaction data according to the transaction data structure tree to obtain transaction data to be analyzed, including: determining one or more target metadata to be obtained based on the computing task; parsing the transaction data to generate a first data frame, wherein the second data frame includes the total load data of the transaction; based on the first data frame, selecting the first header and / or transaction set array of the total load data of each transaction on the blockchain node as the second data frame; locating the position of the target metadata in the second data frame according to the structure tree, and extracting the target metadata from the position as the transaction data to be analyzed.

[0015] According to one embodiment of this disclosure, locating the target metadata from the second data frame according to the structure tree and extracting the target metadata from the location includes: extracting the channel header and signature header from the first header according to the structure tree as a type of target metadata; or, determining the action type array from the transaction set array according to the structure tree, and distinguishing the second header and sub-payload data in the action type array; extracting chaincode proposal payload data and action data from the sub-payload data, and obtaining the transaction's chaincode information based on the chaincode proposal payload data as a type of target metadata; or, extracting the proposal response payload data from the action data, and obtaining the read / write set generated by the transaction during the simulated chaincode execution in the endorsement phase from the proposal response payload data; wherein the data to be analyzed includes at least one of the following target metadata: channel header, signature header, second header, chaincode information, and read / write set.

[0016] According to one embodiment of this disclosure, before executing pre-written computational logic code on the data to be analyzed for all transactions on a blockchain node and obtaining the first computational result, the method further includes: receiving computational logic code sent by the main computing node.

[0017] According to one embodiment of this disclosure, the method, executed by the master computing node in a distributed computing cluster, includes:

[0018] The first aspect of this disclosure proposes a data processing method based on a distributed ledger, executed by a master computing node in a distributed computing cluster, comprising: sending a computing task to a slave computing node in the distributed computing cluster to instruct the slave computing node to perform calculations on the ledger data of the associated blockchain node; receiving a first calculation result of the ledger data reported by the slave computing node; and summarizing the first calculation results reported by each slave computing node to obtain a second calculation result.

[0019] According to one embodiment of this disclosure, the method further includes: receiving pre-written computational logic code and distributing the computational logic code to slave computing nodes in a distributed computing cluster.

[0020] To achieve the above objectives, a second aspect of this disclosure provides a data processing apparatus based on a distributed ledger, executed by a slave computing node in a distributed computing cluster, comprising: a receiving module for receiving a computing task sent by a master computing node in the distributed computing cluster; a loading module for determining the blockchain node associated with the slave computing node and loading ledger data on the blockchain node; a computing module for performing calculations on the ledger data to generate a first calculation result; and a summarizing module for sending the first calculation result to the master computing node for summarization to obtain a second calculation result.

[0021] To achieve the above objectives, a second aspect of this disclosure provides a data processing apparatus based on a distributed ledger, executed by a master computing node in a distributed computing cluster, comprising: a sending module for sending computing tasks to slave computing nodes in the distributed computing cluster, instructing the slave computing nodes to perform calculations on ledger data of associated blockchain nodes; a receiving module for receiving a first calculation result of the ledger data reported by the slave computing nodes; and a transmission module for summarizing the first calculation results reported by each slave computing node to obtain a second calculation result.

[0022] To achieve the above objectives, a third aspect of this disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to implement the distributed ledger-based data processing method as described in the first aspect of this disclosure.

[0023] To achieve the above objectives, a fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the distributed ledger-based data processing method as described in the first aspect of this disclosure.

[0024] To achieve the above objectives, a fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, is used to implement the distributed ledger-based data processing method as described in the first aspect of this disclosure.

[0025] By binding computing nodes and blockchain nodes together, performing single-node computation, and then aggregating the results through a master computing node, it is possible to achieve a holistic analysis of the ledger data on the blockchain nodes, thereby improving the quality and usability of the data processing results. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a data processing method based on a distributed ledger according to one embodiment of the present disclosure;

[0027] Figure 2 This is a schematic diagram of the structure of a distributed computing cluster for a data processing method based on a distributed ledger according to one embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram of another data processing method based on a distributed ledger according to one embodiment of the present disclosure;

[0029] Figure 4 This is a schematic diagram of another data processing method based on a distributed ledger according to one embodiment of the present disclosure;

[0030] Figure 5 This is a schematic diagram of another data processing method based on a distributed ledger according to one embodiment of the present disclosure;

[0031] Figure 6 This is a schematic diagram of a data processing apparatus based on a distributed ledger according to one embodiment of the present disclosure;

[0032] Figure 7 This is a schematic diagram of another distributed ledger-based data processing apparatus according to one embodiment of the present disclosure;

[0033] Figure 8 This is a schematic diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0034] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0035] The acquisition, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0036] Figure 1 This is a schematic diagram illustrating an exemplary implementation of a distributed ledger-based data processing method proposed in this disclosure, as shown below. Figure 1 As shown, this distributed ledger-based data processing method includes the following steps:

[0037] S101 receives computing tasks sent by the master computing node in the distributed computing cluster.

[0038] In this embodiment of the disclosure, the distributed computing cluster is as follows: Figure 2 As shown, there are three types of nodes: master computing nodes, slave computing nodes, and blockchain nodes. Distributed computing clusters can rely on the processors of terminal devices, such as those in computers and mobile phones, or they can rely on servers; no specific limitations are made here.

[0039] It should be noted that there are multiple types of master compute nodes; for example, it can be a Spark compute node. Spark is a popular in-memory distributed computing framework for big data, characterized by its high speed, ease of use, versatility, and diverse operating modes.

[0040] It should be noted that communication connections can be established between the master computing nodes. These connections can be wired or wireless, and no specific limitation is made here. Through these communication connections, computing tasks can be distributed among each master computing node for processing, thereby improving the efficiency of task completion.

[0041] It should be noted that computing tasks can be user-issued or data tasks that need to be processed during operation; no restrictions are imposed here.

[0042] It should be noted that the calculation task can be of various types. For example, it can be a task to assign data quality reduction or a task to assign data calculation. There are no restrictions here, and the specific task depends on the actual situation.

[0043] S102, determine the blockchain node associated with the computing node, and load the ledger data on the blockchain node.

[0044] In this embodiment of the disclosure, the computing node can be of various types. For example, the computing node can be a Spark computing node.

[0045] In this disclosed embodiment, the blockchain node is an innovative integration of various existing technologies, including cryptography, distributed storage, consensus mechanisms, and peer-to-peer (P2P) networks. This combination of technologies endows the blockchain with traceability, tamper-proof properties, and security, cleverly solving the problem of trusted multi-party collaboration. Therefore, blockchain is currently widely used in finance, government affairs, and supply chain fields. A blockchain can be viewed as a database; while traditional databases solve information management problems within an organization, blockchain can solve the problems of trusted data management, sharing, and efficient collaboration between organizations.

[0046] It should be noted that ledger data, also known as blockchain data, is stored in the file system as files. Ledger data files are byte stream files, serialized using the Protobuf serialization protocol. The filenames are prefixed with "blockfile_" (the block file storage format), and the files are stored in a predefined path.

[0047] It should be noted that, as Figure 2As shown, there is a connection between the compute nodes and the blockchain nodes. In this embodiment, the connection can be established by converting the ledger data into a format that the compute nodes can process. For example, the Spark database supports processing data in JavaScript Object Notation (JSON) format and can read JSON files converted from the ledger for analysis and computation. Therefore, a data channel can also be established between Fabric and Spark by converting the ledger into JSON files. Each ledger node in the Fabric blockchain network has complete ledger data, which is stored in a binary file named Blockfile. According to step 2, the Blockfile file can be converted into a JSON format file. Since each host has Fabric blockchain nodes and Spark compute nodes installed, computation tasks can be distributed to each compute node through the master node in the Spark compute cluster. Each compute node can simultaneously perform statistical analysis on the JSON format block file. Each node analyzes a portion of the block file, performs the calculations, and then aggregates the results to the master node, which then outputs the computation results.

[0048] S103, perform calculations on the ledger data to generate the first calculation result.

[0049] In this embodiment of the disclosure, ledger data can be calculated using computing nodes. Before calculating the ledger data, data processing can be performed. This data processing can take many forms, which are not limited here. For example, the ledger data can be packaged, and the data format can be standardized to improve the efficiency and accuracy of the ledger data calculation.

[0050] S104, the first calculation result is sent to the main computing node for aggregation to obtain the second calculation result.

[0051] In this embodiment of the disclosure, after the first calculation result is obtained from the computing node, the first calculation result needs to be sent to the main computing node. In this way, the main computing node summarizes all the first calculation results and performs data analysis and processing to obtain the second calculation result.

[0052] It should be noted that the second calculation result can be either the data itself or an analysis of the data distribution; no limitations are imposed here. For example, the analysis results of blockchain data in a blockchain-based agricultural traceability project can be obtained by analyzing statistical results based on smart contracts, contract functions, and endorsing organizations. For instance, analyzing the number of times smart contracts and contract functions are called, and the number of times endorsing organizations endorse, can provide a clear understanding of the blockchain data distribution of the entire blockchain-based agricultural traceability project.

[0053] In this embodiment, a computing task is first received from the master computing node in the distributed computing cluster. Then, the associated blockchain node is determined, and ledger data on the blockchain node is loaded. The ledger data is then computed to generate a first computation result. Finally, the first computation result is sent to the master computing node for aggregation, resulting in a second computation result. This method of binding the master computing node and the blockchain node, performing single computations, and then aggregating the results through the master computing node enables comprehensive analysis of the ledger data on the blockchain node, improving the quality and usability of the data processing results.

[0054] It should be noted that to determine the blockchain node associated with the computing node, one can first determine the host deployed on the computing node from the host cluster, and then determine the blockchain node associated with the computing node from the blockchain cluster based on the host deployed on the computing node. In this case, each host in the host cluster has one blockchain node deployed on it.

[0055] In this embodiment of the disclosure, the host cluster is a cluster composed of multiple hosts. Each host may be equipped with a distributed computing cluster. Communication connections may be established between the hosts, which may be wired or wireless, without any limitation, and the communication connections are used to realize data interaction between the hosts.

[0056] In the above embodiments, before loading the ledger data on the blockchain node, it is also possible to... Figure 3 To further explain, the method includes:

[0057] S301, call the format conversion program, which obtains the first format ledger data stored on the blockchain node.

[0058] In this embodiment of the disclosure, the ledger data is stored in a first format within a blockchain node, which may be a blockfile file.

[0059] It should be noted that the format conversion program is pre-set and stored in the electronic device's storage space for easy retrieval and use when needed, thereby improving the efficiency of data processing. In this embodiment, the format conversion program may be Protobuf.

[0060] S302, desequence and format conversion are performed on the ledger data in the first format to obtain ledger data in the second format that can be loaded from the computing node.

[0061] In this embodiment of the disclosure, the second format may be JSON format.

[0062] Since the ledger data file (blockfile) is a serialized binary file, it needs to be deserialized before the original data can be obtained. In this embodiment, the storage address of the ledger data in the first format can be obtained, and the ledger data in the first format can be read from the blockchain node based on the storage address.

[0063] Protobuf encoding uses proto files for serialization and deserialization. These proto files define the data structure, i.e., how the data is encoded. Most proto files related to the transaction data structure are located in the Fabric / protos / common / and Fabric / protos / peer directories of the Fabric source code. Based on the source code, the transaction data structure is organized. Using this data structure, ledger files serialized using the Protobuf protocol can be easily converted into JSON format data.

[0064] In this embodiment, a format conversion program is first invoked. This program obtains ledger data in a first format stored on the blockchain node, then deserializes and converts the first-format ledger data to obtain second-format ledger data that can be loaded from the computing node. Thus, the format conversion program converts the first-format ledger data into a second-format ledger data that can be processed by the computing node, thereby establishing the connection between the computing node and the blockchain node and providing a foundation for the computing node to process ledger data.

[0065] In the above embodiments, calculations are performed on the ledger data to generate a first calculation result, and further calculations can be performed using... Figure 4 To further explain, the method includes:

[0066] S401: Based on the ledger data, obtain the transaction data for each transaction.

[0067] In this embodiment of the disclosure, transaction data for each transaction can be obtained by parsing the ledger data. It should be noted that the transaction data may include various elements, such as the number, type, time, and object of the transaction, etc., without any limitation here.

[0068] S402, according to the transaction data structure tree, extract metadata from the transaction data to obtain the transaction data to be analyzed.

[0069] In this embodiment of the disclosure, one or more target metadata that need to be acquired can first be determined based on the computing task. Then, the transaction data is parsed to generate a first data frame, wherein the second data frame includes the total load data of the transaction. Then, based on the first data frame, the first header of the total load data of each transaction on the blockchain node and / or the transaction set array are selected as the second data frame. Finally, according to the structure tree, the location of the target metadata is located in the second data frame, and the target metadata is extracted from the location as the transaction data to be analyzed.

[0070] It should be noted that, according to the structure tree, to locate the target metadata in the second data frame and extract the target metadata from the location, one can first extract the channel header and signature header from the first header as a type of target metadata, or, according to the structure tree, determine the action type array from the transaction set array, distinguish the second header and sub-payload data in the action type array, then extract the chaincode proposal payload data and action data from the sub-payload data, and obtain the transaction's chaincode information based on the chaincode proposal payload data as a type of target metadata, or extract the proposal response payload data from the action data, and obtain the read / write set generated by the transaction during the simulated chaincode execution in the endorsement phase from the proposal response payload data.

[0071] The data to be analyzed includes at least one of the following target metadata: channel header, signature header, second header, chaincode information, and read / write set.

[0072] It's worth noting that the header can be further divided into the channel header and the signature header. These two headers contain information such as timestamps, transaction accounts, and MSPIDs, providing a wealth of information that can help us analyze some of the transaction's metadata and can be used as analysis objects. Next, let's look at the payload's data section. This section consists of an array of action types, as each transaction endorsement forms an action object. Actions can also be divided into headers and payloads. The headers contain relatively little information; the focus is on the payload, which consists of chaincode_proposal_payload and actions. First, let's examine the chaincode_proposal_payload section, which consists of input and a TransientMap. The input is further composed of chaincode_spec, which contains information such as chaincode_id, timeout, and input, and can be used as an analysis object to help us analyze the chaincode invoked in this transaction. Next, returning to the action section, it consists of proposal_response_payload (the payload data of the proposal response) and endorsements (endorsement information). The focus is on proposal_response_payload, which can be broken down into extension and proposal_hash. The extension contains a significant amount of information, organized by results, chaincode_id, events, etc. Most importantly, results contains rwset (read-write set), which is generated during the simulated execution of chaincode in the endorsement phase. This represents which data was read in this transaction and which data will be changed. This part is also worth analyzing and can be considered an analytical object.

[0073] S403 executes pre-written computational logic code on the data to be analyzed for all transactions on the blockchain node to obtain the first computational result.

[0074] It should be noted that before executing the pre-written computational logic code on all transaction data to be analyzed on the blockchain node and obtaining the first computational result, it is also necessary to receive computational logic code sent by the main computing node. In this way, the pre-written computational logic code can be used to process the data to be analyzed into the desired first computational result.

[0075] It should be noted that the calculation logic code can be modified according to actual design needs, and no restrictions are imposed here. For example, the calculation logic code may include the following: ① select(): This operation selects data from a subset of columns in a DataFrame, and can also rename columns, similar to the select operation in SQL. ② groupBy(): This operation is used to group records, which can be used for categorized statistics. ③ count(): This operation is used to count the number of records. ④ explode(): This operation expands a given array or map into new rows. This function can be used to expand nested structures in a JSON file.

[0076] Figure 5 This is a schematic diagram illustrating an exemplary implementation of a distributed ledger-based data processing method proposed in this disclosure, executed by the master computing node in a distributed computing cluster, such as... Figure 5 As shown, this distributed ledger-based data processing method includes the following steps:

[0077] S501 sends a computation task to a compute node in a distributed computing cluster, instructing the compute node to perform computations on the ledger data of the associated blockchain node.

[0078] For specific steps, please refer to the above embodiments, which will not be repeated here.

[0079] S502 receives the first calculation result of the ledger data reported from the computing node.

[0080] For specific steps, please refer to the above embodiments, which will not be repeated here.

[0081] S503, summarize the first calculation results reported from each computing node to obtain the second calculation result.

[0082] For specific steps, please refer to the above embodiments, which will not be repeated here.

[0083] In this embodiment, a computation task is first sent to a slave computing node in the distributed computing cluster, instructing the slave computing node to perform computations on the ledger data of the associated blockchain node. Then, the first computation result of the ledger data reported by the slave computing node is received. Finally, the first computation results reported by each slave computing node are summarized to obtain a second computation result. By binding slave computing nodes and blockchain nodes, performing single computations, and summarizing through the master computing node, a holistic analysis of the ledger data on the blockchain node can be achieved, improving the quality and usability of the data processing results.

[0084] Corresponding to the distributed ledger-based data processing methods provided in the above embodiments, one embodiment of this disclosure also provides a distributed ledger-based data processing apparatus. Since the distributed ledger-based data processing apparatus provided in this disclosure corresponds to the distributed ledger-based data processing methods provided in the above embodiments, the implementation methods of the above distributed ledger-based data processing methods are also applicable to the distributed ledger-based data processing apparatus provided in this disclosure, and will not be described in detail in the following embodiments.

[0085] Figure 6 This is a schematic diagram of a data processing device based on a distributed ledger proposed in this disclosure, as shown below. Figure 6 As shown, the distributed ledger-based data processing device 600 includes: a receiving module 610, a loading module 620, a calculation module 630, and a summarizing module 640.

[0086] The receiving module 610 is used to receive computing tasks sent by the master computing node in the distributed computing cluster.

[0087] The loading module 620 is used to determine the blockchain node associated with the computing node and load the ledger data on the blockchain node.

[0088] The calculation module 630 is used to perform calculations on the ledger data and generate a first calculation result.

[0089] The aggregation module 640 is used to send the first calculation result to the main computing node for aggregation to obtain the second calculation result.

[0090] In one embodiment of this disclosure, the loading module 620 is further configured to: determine the host deployed by the slave computing node from the host cluster; and determine the blockchain node associated with the slave computing node from the blockchain cluster based on the host deployed by the slave computing node, wherein each host in the host cluster has a blockchain node deployed in the blockchain cluster.

[0091] In one embodiment of this disclosure, the loading module 620 is further configured to: call a format conversion program, the format conversion program obtaining ledger data in a first format stored on the blockchain node; and deserialize and convert the ledger data in the first format to obtain ledger data in a second format that can be loaded from the computing node.

[0092] In one embodiment of this disclosure, the loading module 620 is further configured to: obtain the storage address of the ledger data in the first format, and read the ledger data in the first format from the blockchain node according to the storage address.

[0093] In one embodiment of this disclosure, the computing module 630 is further configured to: obtain transaction data for each transaction based on ledger data; extract metadata from the transaction data according to the transaction data structure tree to obtain transaction data to be analyzed; and execute pre-written computing logic code on the transaction data to be analyzed for all transactions on the blockchain node to obtain a first computing result.

[0094] In one embodiment of this disclosure, the computing module 630 is further configured to: determine one or more target metadata to be acquired based on the computing task; parse the transaction data to generate a first data frame, wherein the second data frame includes the total load data of the transaction; based on the first data frame, select the first header and / or transaction set array of the total load data of each transaction on the blockchain node as the second data frame; locate the location of the target metadata in the second data frame according to the structure tree, and extract the target metadata from the location as the transaction data to be analyzed.

[0095] In one embodiment of this disclosure, the computing module 630 is further configured to: extract a channel header and a signature header from a first header according to a structure tree, as a type of target metadata; or, determine an array of action types from a transaction set array according to a structure tree, and distinguish a second header and sub-payload data in the array of action types; extract chaincode proposal payload data and action data from the sub-payload data, and obtain the chaincode information of the transaction according to the chaincode proposal payload data, as a type of target metadata; or, extract the payload data of the proposal response from the action data, and obtain the read / write set generated by the transaction when simulating chaincode execution in the endorsement phase from the payload data of the proposal response; wherein the data to be analyzed includes at least one of the following target metadata: channel header, signature header, second header, chaincode information, and read / write set.

[0096] In one embodiment of this disclosure, before executing pre-written computational logic code on the data to be analyzed for all transactions on the blockchain node and obtaining the first computational result, the method further includes: receiving computational logic code sent by the main computing node.

[0097] Figure 7 This is a schematic diagram of a data processing device based on a distributed ledger proposed in this disclosure, which is executed by the master computing node in a distributed computing cluster, such as... Figure 7 As shown, the distributed ledger-based data processing device 700 includes: a sending module 710, a receiving module 720, and a transmission module 730.

[0098] The sending module 710 is used to send computing tasks to the slave computing nodes in the distributed computing cluster, so as to instruct the slave computing nodes to perform calculations on the ledger data of the associated blockchain nodes.

[0099] The receiving module 720 is used to receive the first calculation result of the ledger data reported from the computing node.

[0100] The transmission module 730 is used to summarize the first calculation results reported by each computing node to obtain the second calculation result.

[0101] In one embodiment of this disclosure, the sending module 710 is further configured to: receive pre-written computational logic code and distribute the computational logic code to the slave computing nodes in the distributed computing cluster.

[0102] To implement the above embodiments, this disclosure also proposes an electronic device 800, such as... Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802 communicatively connected to the processor. The memory 802 stores instructions executable by at least one processor. The instructions are executed by at least one processor 801 to implement the distributed ledger-based data processing method as described in the first aspect of this disclosure.

[0103] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to implement the distributed ledger-based data processing method as described in the first aspect of this disclosure.

[0104] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program that, when executed by a processor, implements the distributed ledger-based data processing method as described in the first aspect of this disclosure.

[0105] In the description of this disclosure, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0106] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise expressly specified.

[0107] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0108] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A data processing method based on distributed ledger, characterized in that, Executed by a slave compute node in a distributed computing cluster, the method includes: Receive computing tasks sent by the master computing node in the distributed computing cluster; The blockchain node associated with the computing node is determined, and the ledger data on the blockchain node is loaded, wherein the ledger data is blockchain data and is stored in the file system in the form of a file. The ledger data is calculated to generate a first calculation result; The first calculation result is sent to the main computing node for aggregation to obtain the second calculation result; The step of calculating the ledger data to generate a first calculation result includes: Based on the ledger data, obtain the transaction data for each transaction; Based on the structure tree of the transaction data, metadata is extracted from the transaction data to obtain the data to be analyzed for the transaction; For the data to be analyzed from all transactions on the blockchain node, execute pre-written computational logic code to obtain the first computational result.

2. The method according to claim 1, characterized in that, The process of determining the blockchain node associated with the computing node includes: From the host cluster, determine the hosts deployed on the compute nodes; Based on the host deployed on the slave computing node, the blockchain node associated with the slave computing node is determined from the blockchain cluster, wherein each host in the host cluster has one blockchain node deployed in the blockchain cluster.

3. The method according to claim 1, characterized in that, Before loading the ledger data on the blockchain node, the process also includes: The format conversion program is invoked to obtain ledger data in a first format stored on the blockchain node; The ledger data in the first format is deserialized and converted to obtain ledger data in the second format that can be loaded from the computing node.

4. The method according to claim 3, characterized in that, The format conversion program obtains ledger data in a first format stored on the blockchain node, including: Obtain the storage address of the ledger data in the first format, and read the ledger data in the first format from the blockchain node according to the storage address.

5. The method according to claim 1, characterized in that, The step of extracting metadata from the transaction data according to the transaction data structure tree to obtain the transaction data to be analyzed includes: Based on the computational task, determine one or more target metadata that need to be acquired; The transaction data is parsed to generate a first data frame, wherein the first data frame includes the total load data of the transaction; Based on the first data frame, the first header and / or transaction set array of the total load data of each transaction on the blockchain node are selected as the second data frame; According to the structure tree, the location of the target metadata is located in the second data frame, and the target metadata is extracted from the location as the data to be analyzed in the transaction.

6. The method according to claim 5, characterized in that, The step of locating the target metadata from the second data frame according to the structure tree and extracting the target metadata from the location includes: According to the structure tree, the channel header and signature header are extracted from the first header as a type of target metadata; or... According to the structure tree, determine the array of action types from the array of transaction sets, and distinguish the second header and sub-payload data in the array of action types; Extract chaincode proposal payload data and action data from the sub-payload data, and obtain the chaincode information of the transaction based on the chaincode proposal payload data, as a type of target metadata; or, Extract the payload data of the proposal response from the action data, and obtain the read-write set generated by the transaction when simulating chaincode execution in the endorsement phase from the payload data of the proposal response; The data to be analyzed includes at least one of the following target metadata: the channel header, the signature header, the second header, the chaincode information, and the read / write set.

7. The method according to claim 1, characterized in that, Before executing pre-written computational logic code on the data to be analyzed for all transactions on the blockchain node and obtaining the first computation result, the method further includes: receiving the computational logic code sent by the main computing node.

8. A data processing method based on distributed ledger, characterized in that, The method, executed by the master computing node in a distributed computing cluster, includes: Send a computing task to the slave computing node in the distributed computing cluster to instruct the slave computing node to perform calculations on the ledger data of the associated blockchain node in accordance with the method described in any one of claims 1-7; Receive the first calculation result of the ledger data reported from the computing node; The first calculation results reported by each computing node are summarized to obtain the second calculation result.

9. The method according to claim 8, characterized in that, The method further includes: Receive pre-written computational logic code and distribute the computational logic code to the slave computing nodes in the distributed computing cluster.

10. A data processing device based on a distributed ledger, characterized in that, Executed by slave compute nodes in a distributed computing cluster, including: The receiving module is used to receive computing tasks sent by the master computing node in the distributed computing cluster; A loading module is used to determine the blockchain node associated with the computing node and load the ledger data on the blockchain node, wherein the ledger data is blockchain data and is stored in the file system in the form of a file; The calculation module is used to calculate the ledger data and generate a first calculation result; The aggregation module is used to send the first calculation result to the main computing node for aggregation to obtain the second calculation result; The step of calculating the ledger data to generate a first calculation result includes: Based on the ledger data, obtain the transaction data for each transaction; Based on the structure tree of the transaction data, metadata is extracted from the transaction data to obtain the data to be analyzed for the transaction; For the data to be analyzed from all transactions on the blockchain node, execute pre-written computational logic code to obtain the first computational result.

11. A data processing device based on a distributed ledger, characterized in that, Executed by the master compute node in the distributed computing cluster, including: The sending module is used to send computing tasks to the slave computing nodes in the distributed computing cluster, instructing the slave computing nodes to perform calculations on the ledger data of the associated blockchain nodes in accordance with the method described in any one of claims 1-7; A receiving module is configured to receive the first calculation result of the ledger data reported from the computing node; The transmission module is used to summarize the first calculation results reported by each computing node to obtain the second calculation result.

12. An electronic device, characterized in that, Including memory and processor; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7 or 8-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7 or 8-9.

Images