Blockchain-based data processing method, apparatus, device, medium, and program product

Abstract

This disclosure provides a blockchain-based data processing method applicable to the field of blockchain technology. The method includes: executing an inbound smart contract based on a business blockchain to write collected source data to a decentralized database; executing a data processing smart contract based on the business blockchain to process the source data in the decentralized database to form outbound data, and performing the write operation on the outbound data to write it to the decentralized database; executing a file generation smart contract based on the business blockchain to generate an outbound file; and transmitting the outbound file to a target terminal. This disclosure also provides a blockchain-based data processing system, device, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the field of blockchain technology, specifically to a blockchain data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the increasing sophistication of my country's financial system and the rapid increase in operational risks, coupled with the impact of internet finance and big data, the analysis of banks' business data has become particularly important. As technologies such as data lakes continue to develop and mature, banks are placing higher demands on the analysis of customer information and transaction data.

[0003] Traditional bank data regulatory reporting uses a centralized data lake to store source data, and then a big data processing engine cleans and processes this source data according to the data standards of specific business scenarios to form regulatory indicator data and detailed data. This data is then transmitted to the institutions corresponding to the specific business scenarios.

[0004] In realizing the concept disclosed herein, the inventors discovered that the related technologies have at least the following problems: centralized clusters are prone to failure, data processing is not timely, data is incomplete, and metadata nodes are vulnerable to attacks. Summary of the Invention

[0005] In view of the above problems, this disclosure provides blockchain-based data processing methods, apparatuses, devices, media, and program products to improve the security and efficiency of data processing.

[0006] According to a first aspect of this disclosure, a blockchain-based data processing method is provided, comprising: executing an inbound smart contract based on a business blockchain to perform a write operation on collected source data and write it to a decentralized database; executing a data processing smart contract based on the business blockchain to process the source data in the decentralized database to form outbound data, and performing the write operation on the outbound data to write it to the decentralized database; executing a file generation smart contract based on the business blockchain to generate an outbound file; and transmitting the outbound file to a target end.

[0007] According to an embodiment of this disclosure, the source data and the outgoing data are inbound data, and the write operation includes: separating the inbound data into a first preset number of sub-data blocks; and using erasure coding to encode the first preset number of data blocks into a second preset number of data blocks, wherein the second preset number of data blocks has a third preset number of redundant data blocks compared to the first preset number of sub-data blocks.

[0008] According to an embodiment of this disclosure, after the first preset number of data blocks are written into a second preset number of data blocks using erasure coding, the write operation further includes: identifying the content in the sub-data blocks based on multiple hashes; and storing the identified sub-data blocks into a decentralized node based on a distributed hash table.

[0009] According to embodiments of this disclosure, the execution of the smart contract for entering the lake based on the business blockchain includes: periodically executing the smart contract for entering the lake based on a preset collection time.

[0010] According to embodiments of this disclosure, the decentralized database includes a first decentralized database and a second decentralized database, wherein the source data is written to the first decentralized database and the outgoing data is written to the second decentralized database.

[0011] According to embodiments of this disclosure, the step of executing a data processing smart contract based on the business blockchain to process source data in the decentralized database to form outgoing data, and storing the outgoing data in the decentralized database, includes: performing a mapping operation on the source data in a first decentralized node to obtain data to be processed, wherein the first decentralized node is a node in the first decentralized database, and the mapping operation is used to rearrange data blocks; temporarily storing the data to be processed in a second decentralized node, wherein the second decentralized node is a node in the first decentralized database and is an adjacent node of the first decentralized node; performing a simplification operation on the data to be processed on the second decentralized node based on a preset calculation logic to obtain the outgoing data; and storing the outgoing data in a third decentralized node, wherein the third decentralized node is a node in the second decentralized database.

[0012] According to embodiments of this disclosure, after the source data is written to the decentralized database, or after the outgoing data is written to the decentralized database, the method further includes: monitoring the entry status of the source data and / or monitoring the entry status of the outgoing data; if the entry status is successful, issuing an entry success command; and if the entry status is unsuccessful, issuing an entry failure command to complete the alarm.

[0013] A second aspect of this disclosure provides a blockchain-based data processing device, comprising: a data acquisition module, a data processing module, a file generation module, and a file transmission module. The data acquisition module is configured to execute an inbound smart contract based on a business blockchain to perform a write operation on the acquired source data, writing it to a decentralized database. The data processing module is configured to execute a data processing smart contract based on the business blockchain to process the source data in the decentralized database to form outbound data, and to perform the write operation on the outbound data, writing it to the decentralized database. The file generation module is configured to execute a file generation smart contract based on the business blockchain to generate an outbound file. The file transmission module is configured to transmit the outbound file to a target end.

[0014] According to an embodiment of this disclosure, the source data and the outgoing data are inbound data, and the write operation includes: separating the inbound data into a first preset number of sub-data blocks; and using erasure coding to encode the first preset number of data blocks into a second preset number of data blocks, wherein the second preset number of data blocks has a third preset number of redundant data blocks compared to the first preset number of sub-data blocks.

[0015] According to an embodiment of this disclosure, after the first preset number of data blocks are written into a second preset number of data blocks using erasure coding, the write operation further includes: identifying the content in the sub-data blocks based on multiple hashes; and storing the identified sub-data blocks into a decentralized node based on a distributed hash table.

[0016] According to embodiments of this disclosure, the data acquisition module is further configured to periodically execute the lake entry smart contract based on a preset acquisition time.

[0017] According to embodiments of this disclosure, the decentralized database includes a first decentralized database and a second decentralized database, wherein the source data is written to the first decentralized database and the outgoing data is written to the second decentralized database.

[0018] According to embodiments of this disclosure, the data processing module is further configured to: perform a mapping operation on the source data in the first decentralized node to obtain data to be processed, wherein the first decentralized node is a node in the first decentralized database, and the mapping operation is used to rearrange data blocks; temporarily store the data to be processed in a second decentralized node, wherein the second decentralized node is a node in the first decentralized database and is an adjacent node of the first decentralized node; perform a simplification operation on the data to be processed on the second decentralized node based on preset calculation logic to obtain the outgoing data; and store the outgoing data in a third decentralized node, wherein the third decentralized node is a node in the second decentralized database.

[0019] According to an embodiment of this disclosure, the device further includes a monitoring and alarm module, wherein the monitoring module is used to monitor the warehousing status of the source data and / or monitor the warehousing status of the outgoing data; if the warehousing status is successful, a warehousing success command is issued; and if the warehousing status is unsuccessful, a warehousing failure command is issued to complete the alarm.

[0020] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the aforementioned blockchain-based data processing method.

[0021] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the aforementioned blockchain-based data processing method.

[0022] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned blockchain-based data processing method. Attached Figure Description

[0023] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0024] Figure 1 The diagram illustrates an application scenario of a blockchain-based data processing method according to an embodiment of the present disclosure.

[0025] Figure 2 A flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0026] Figure 3AA flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0027] Figure 3B A flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0028] Figure 4 A flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0029] Figure 5 The illustration schematically depicts an architecture combining Hadoop, a big data analytics technology, with IPFS, a decentralized storage technology, according to an embodiment of this disclosure.

[0030] Figure 6 This illustration schematically depicts an embodiment of the present disclosure in... Figure 5 A schematic diagram of IPFS data processing architecture, a decentralized storage technology.

[0031] Figure 7 The diagram illustrates a complete flowchart of a blockchain-based data processing method according to an embodiment of the present disclosure.

[0032] Figure 8 A schematic block diagram of a blockchain-based data processing apparatus according to an embodiment of the present disclosure is shown.

[0033] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a blockchain-based data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0034] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0036] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0037] When using expressions such as "at least one of A, B, and C", the expression should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0038] When using expressions such as "at least one of A, B, and C", the expression should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0039] Before explaining the embodiments of this disclosure in detail, the key technical terms involved in this disclosure will be explained one by one:

[0040] Blockchain: Blockchain is a ledger technology maintained by multiple parties, using cryptography to ensure secure transmission and access, and enabling consistent data storage, tamper-proofing, and non-repudiation. It is also known as distributed ledger technology. A typical blockchain stores data in a block-chain structure.

[0041] Smart contracts: The smart contract layer is responsible for implementing, compiling and deploying the business logic of the blockchain system in the form of code, completing the triggering and automatic execution of the predetermined rules, and minimizing human intervention.

[0042] Banking Regulatory Reporting: The rapid development of China's banking industry has brought about profound changes in the regulatory system and mechanism of China's banking industry. Adhering to the regulatory concept of "risk-based", the China Banking and Insurance Regulatory Commission has planned and constructed a framework for the banking regulatory information system. All banks are required to report relevant data at the prescribed frequency according to the requirements of the regulatory authorities.

[0043] Metadata: Metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0044] Decentralized storage: Decentralized storage is a distributed storage system with the participation of numerous nodes. It can get rid of the limitations of large centralized storage and geographical location. Decentralized storage is more secure, faster, and more cost-effective. Since decentralized storage does not have the concept of metadata, it can prevent data security problems caused by metadata tampering.

[0045] Erasure coding: The principle of erasure coding is to divide the data into K blocks and encode them into K+R encoded data blocks. The complete data can be recovered from any K encoded data blocks. Each erasure coding node only needs to store one of the encoded data blocks, which is 1 / K of the complete data block, thus reducing the storage burden.

[0046] Hadoop: A distributed system infrastructure that allows users to develop distributed programs without understanding the underlying details of distributed systems.

[0047] Hadoop Distributed File System (HDFS) is characterized by high fault tolerance and is designed to be deployed on inexpensive hardware. It also provides high throughput access to application data, making it suitable for applications with very large datasets.

[0048] MapReduce: A programming model for parallel computation of large datasets (greater than 1TB). The concepts of "Map" and "Reduce" are its core ideas, borrowed from functional programming languages, as well as features from vector programming languages. It greatly simplifies the process for programmers without prior knowledge of distributed parallel programming, allowing them to run their programs on distributed systems. Current software implementations specify a Map function to map a set of key-value pairs into a new set of key-value pairs, and specify concurrent Reduce functions to ensure that each mapped key-value pair shares the same key.

[0049] Yet Another Resource Negotiator (YARN) is a new Hadoop resource manager that can improve cluster utilization, resource management uniformity, and data sharing.

[0050] The InterPlanetary File System (IPFS) is a network protocol designed to create persistent and distributed storage and sharing of files. It is a content-addressable peer-to-peer hypermedia distribution protocol. Nodes in the IPFS network constitute a distributed file system. IPFS offers at least the following advantages: 1. Permanent, decentralized storage and sharing of files (storage DHTs in a blockchain model). 2. Identifying files by generating unique hash values ​​based on their content, rather than by their storage location. Files with identical content will only exist once in the system, saving storage space.

[0051] A data lake is a repository that allows the storage of large amounts of raw data, that is, data that has not been prepared, processed, or manipulated according to a specific pattern.

[0052] In the field of data processing, taking bank data regulatory reporting as an example, traditional bank data regulatory reporting uses a centralized data lake to store source data, and uses a big data processing engine to clean and process the source data according to the regulatory reporting data standards to form regulatory indicator data and detailed data. Then, according to the requirements of the regulatory authorities, the above indicator data and detailed data are processed and summarized, and the data to be reported is transmitted to the corresponding regulatory system application of the regulatory authorities. The application then sends the regulatory reporting data to the regulatory authorities through a dedicated line.

[0053] However, the centralized data lake model used in banks' data supervision and reporting systems has at least the following drawbacks:

[0054] 1. Traditional bank data supervision and reporting adopts a centralized data lake model. Although the processing efficiency is relatively high, the centralized cluster is prone to single point of failure. Even if there is a primary and backup cluster, the integrity of the data cannot be guaranteed when a certain cluster fails. Therefore, the operation continuity of the supervision and reporting application cannot be guaranteed, which can easily lead to the risk of untimely supervision and reporting and incomplete data.

[0055] 2. In traditional data lake storage systems, metadata is stored on a central node, which is vulnerable to malicious modification and has poor security. Once the metadata node is attacked, the accuracy of the reported data cannot be guaranteed.

[0056] 3. Centralized data lakes suffer from the problem of data needing to be repeatedly transmitted and stored, resulting in increased storage costs and wasted communication overhead.

[0057] Based on the above points, banks and other regulated institutions may be unable to meet the timeliness and data quality standards required by regulatory authorities.

[0058] To address the problems existing in the prior art, embodiments of this disclosure provide a blockchain-based data processing method, comprising: executing an inbound smart contract based on a business blockchain to perform a write operation on the collected source data and write it to a decentralized database; executing a data processing smart contract based on the business blockchain to process the source data in the decentralized database to form outbound data, and performing the write operation on the outbound data to write it to the decentralized database; executing a file generation smart contract based on the business blockchain to generate an outbound file; and transmitting the outbound file to a target end.

[0059] In the embodiments disclosed herein, data collection and transmission are conducted entirely using blockchain in conjunction with a decentralized database, which greatly ensures data security and avoids the possibility of data tampering during the process. Using blockchain in conjunction with a decentralized database also effectively avoids the inefficiency of relying solely on blockchain for data storage and processing, thus improving the efficiency of the entire data processing workflow.

[0060] Figure 1 The diagram illustrates an application scenario of a blockchain-based data processing method according to an embodiment of the present disclosure.

[0061] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0062] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0063] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0064] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0065] It should be noted that the blockchain-based data processing method provided in this disclosure can generally be executed by server 105. Correspondingly, the blockchain-based data processing device provided in this disclosure can generally be located in server 105. The blockchain-based data processing method provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the blockchain-based data processing device provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0066] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0067] The following will be based on Figure 1 The described scene, through Figures 2-7 The blockchain-based data processing method of the disclosed embodiments is described in detail.

[0068] Figure 2 A flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0069] like Figure 2 As shown, the blockchain-based data processing method of this embodiment includes operations S210 to S240, and the blockchain-based data processing method can be executed by server 105.

[0070] In operation S210, the smart contract for entering the lake is executed based on the business blockchain to perform a write operation on the collected source data and write it to the decentralized database.

[0071] The source data is generated by the bank's relevant business application systems, such as core business systems, payment systems, bank card systems, and credit management systems. Correspondingly, these business application systems typically include modules such as "Customer Information Management," "Core Accounting Processing," "Risk Management," "Public Parameter Management," and "Business Data Storage." These systems enable the provision of relevant financial services to bank customers, including account management, accounting processing, interest rate management, and risk control. During the execution of these financial services, the operation of different modules generates business-related source data, which is then stored.

[0072] For example, the customer information management module is primarily responsible for managing personal and corporate customer information, including the processes of collection, transmission, storage, use, protection, deletion, and destruction. Its core objective is to provide financial services to customers within the bounds of the principle of reasonable use of customer information. The core accounting module is mainly responsible for business functions such as transaction records, accounting, and journal entries, recording transaction details, transfer information, and beginning and ending data related to accounting. The risk management module is primarily responsible for providing early warnings of relevant risks within the business system, such as capital adequacy ratio, non-performing loan ratio, provision coverage ratio, and liquidity ratio. This module provides water level indicators, triggering alarms or taking other measures when certain indicators exceed or fall below the water level. The public parameter management module is primarily responsible for managing relevant parameters of the business system, such as basic parameters, interest rate parameters, fee parameters, product parameters, and transaction parameters. This module enables the definition and modification of parameters by institutional level, regional code, and card BIN, thereby achieving digital management of the banking business system. The business data storage module mainly provides storage capacity to the aforementioned processing modules, primarily storing customer information, accounting data, risk indicators, public parameters, and other data generated or managed by the "Customer Information Management Module," "Core Accounting Processing Module," "Risk Management Module," and "Public Parameter Management Module."

[0073] A smart contract can be understood as a protocol capable of automatically executing tasks. It contains predefined constraints. Smart contracts offer the advantages of uninterrupted, unconditional execution without human intervention. For example, a lake entry smart contract collects data, and its constraints are related to data collection. Similarly, a data processing smart contract processes data, and its constraints are related to data processing.

[0074] Traditionally, databases used for data storage are broadly categorized into two types: centralized databases and distributed databases. We won't elaborate on the former here. However, there are significant differences between decentralized and distributed databases. The most important difference is that distributed databases have a centralized management node that participates in data write and write operations. Decentralized databases, on the other hand, do not have a centralized management node. This prevents unauthorized access to and / or tampering with the centralized management node during database operations, thereby reducing the risk of data leakage and / or tampering.

[0075] Similarly, decentralized databases and blockchains also differ in some ways. When storing data, blockchains use consensus for chain-like storage, while decentralized databases do not require consensus or chain-like storage. Furthermore, blockchains have limited storage space and slow data access, making it difficult to store large files. Therefore, decentralized databases are used to store and process large amounts of data aggregated from various applications.

[0076] Specifically, the business blockchain can be named based on specific banking business. For example, in the business scenario of bank regulatory reporting, the business blockchain can be called the reporting blockchain.

[0077] According to embodiments of this disclosure, the execution of the smart contract for entering the lake based on the business blockchain includes: periodically executing the smart contract for entering the lake based on a preset collection time.

[0078] The preset collection time can be customized by the user based on the specific scenario, which will not be elaborated here.

[0079] For example, the preset data collection time is two hours. Another example is that one application is assigned one blockchain node, and the smart contract specifies that source data from ten applications must be collected, and that the contract is considered successful only if all source data from the ten applications is successfully entered into the database.

[0080] Understandably, the advantage of collecting source data to a decentralized database based on a preset collection time is that it avoids frequent connections between the decentralized database and specific applications, thereby achieving traffic protection.

[0081] In operation S220, a data processing smart contract is executed based on the business blockchain to process the source data in the decentralized database to form outgoing data, and the write operation is performed on the outgoing data to write it into the decentralized database.

[0082] The logic of the constraint rules in the data processing smart contract is determined based on the specific application scenario.

[0083] For example, in the application scenario of bank regulatory reporting, the aforementioned data processing smart contracts are mainly implemented according to the logic of the regulatory authorities' requirements for data, and the source data is processed through these logics.

[0084] According to embodiments of this disclosure, the decentralized database includes a first decentralized database and a second decentralized database, wherein the source data is written to the first decentralized database and the outgoing data is written to the second decentralized database.

[0085] It should be noted that the write / write operation process is the same in both the first and second decentralized databases; the only difference is the content of the stored data.

[0086] Understandably, setting up two decentralized databases allows for the separate storage of unprocessed source data and processed outgoing data, preventing data contamination between different databases.

[0087] Of course, parameters can also be set in decentralized databases, including the number of replicas and fault tolerance. Appropriate parameter settings can determine the storage utilization, fault tolerance, and computational efficiency of the first or second decentralized database mentioned above.

[0088] In operation S230, a smart contract is generated based on the business blockchain execution file to generate an outgoing file.

[0089] Specifically, the exported files are generated in external applications.

[0090] In operation S240, the external file is transmitted to the target terminal.

[0091] For example, in the application scenario of bank regulatory reporting, the target is the bank regulatory agency.

[0092] In the embodiments disclosed herein, data collection and transmission are conducted entirely using blockchain in conjunction with a decentralized database, which greatly ensures data security and avoids the possibility of data tampering during the process. Using blockchain in conjunction with a decentralized database also effectively avoids the inefficiency of relying solely on blockchain for data storage and processing, thus improving the efficiency of the entire data processing workflow.

[0093] Since the write / write-out operations are implemented identically in both the first and second decentralized databases, the write operations in this disclosure will be explained in detail below.

[0094] Figure 3A A flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0095] like Figure 3A As shown, a blockchain-based data processing method in this embodiment includes operations S310 to S320.

[0096] In operation S310, the data to be stored is divided into a first preset number of sub-data blocks.

[0097] The source data and the externally transmitted data are collectively referred to as inbound data.

[0098] In operation S320, erasure coding is used to encode the first preset number of data blocks into a second preset number of data blocks, wherein the second preset number of data blocks have a third preset number of redundant data blocks compared to the first preset number of sub-data blocks.

[0099] Specifically, the first preset quantity corresponds to quantity K, the second preset quantity corresponds to quantity K+R, and the third preset quantity corresponds to quantity R. The principle of erasure coding is to divide the data into K blocks and encode them into K+R encoded data blocks using erasure coding. The complete data can be recovered from any K encoded data blocks. Each erasure coding node only needs to store one of the encoded data blocks, which is 1 / K of the complete data block.

[0100] In the embodiments of this disclosure, any first preset number of encoded data blocks can be used to recover complete data. Each erasure coding node only needs to store one of the encoded data blocks, thus reducing the storage burden.

[0101] Figure 3B A flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0102] like Figure 3B As shown, in addition to operations S310 to S320, the blockchain-based data processing method of this embodiment also includes operations S330 to S340.

[0103] In operation S330, the contents of the sub-data block are identified based on multiple hashes.

[0104] In operation S340, the identified sub-data blocks are stored in the decentralized node based on the distributed hash table.

[0105] Based on an accessible routing table, peer-to-peer communication between different decentralized nodes is achieved through node management, enabling support for distributed hash tables (DHT) and content addressing.

[0106] It is understood that the above operations S310 to S320 and S330 to S340 are applicable to the first decentralized database and the second decentralized database.

[0107] In the embodiments disclosed herein, each storage node provides storage capacity, and the stored data structure is mainly MerkleDAG. This data structure uses multiple hashes to uniquely identify the content of a data block. In addition, the hash value can be easily checked to confirm whether the data has been tampered with, thus achieving the purpose of preventing attacks. At the same time, since data blocks with the same content have the same hash, duplicate data can be easily removed, saving storage space.

[0108] Figure 4 A flowchart illustrating a blockchain-based data processing method according to an embodiment of the present disclosure is shown.

[0109] Figure 5 The illustration schematically depicts an architecture combining Hadoop, a big data analytics technology, with IPFS, a decentralized storage technology, according to an embodiment of this disclosure.

[0110] Figure 6 This illustration schematically depicts an embodiment of the present disclosure in... Figure 5 A schematic diagram of IPFS data processing architecture, a decentralized storage technology.

[0111] like Figure 5 As shown, a node is equipped with the big data analytics software Hadoop and the decentralized storage database IPFS. The big data analytics software Hadoop includes functional components such as MapReduce and Yam.

[0112] The MapReduce component, a submodule of the big data analytics software Hadoop, is primarily used for data processing and analysis to achieve big data mining. The Yam component, also a submodule of Hadoop, is mainly used for node resource management, enabling the MapReduce component to better utilize resources such as CPU and memory. IPFS, a decentralized database, is a decentralized storage technology, a file-based database that lacks a centralized metadata management node and uses techniques such as hash addressing for data retrieval.

[0113] By enabling MapReduce components to directly read data from the IPFS decentralized database on the decentralized nodes they are running on (usually adjacent ones), and by using network connection components to enable connections and communication between different MapReduce components and IPFS decentralized databases, reading data from the nearest node can greatly reduce the bandwidth occupied by data transmission between different nodes.

[0114] Combination Figure 4 and Figure 6 As shown, a blockchain-based data processing method in this embodiment includes operations S410 to S440. Operations S410 to S440 correspond to the aforementioned operation S220.

[0115] In operation S410, a mapping operation is performed on the source data in the first decentralized node to obtain data to be processed, wherein the first decentralized node is a node in the first decentralized database, and the mapping operation is used to rearrange data blocks.

[0116] Specifically, the mapping operation is called Map.

[0117] In operation S420, the data to be processed is temporarily stored in a second decentralized node, wherein the second decentralized node is a node in the first decentralized database and is an adjacent node of the first decentralized node.

[0118] In operation S430, based on preset calculation logic, the data to be processed on the second decentralized node is simplified to obtain the data to be sent out.

[0119] Specifically, the simplification operation is called Reduce.

[0120] In operation S440, the outgoing data is stored in a third decentralized node, wherein the third decentralized node is a node in the second decentralized database.

[0121] Taking bank regulatory reporting data as an example, the first decentralized database is Figure 6 The source data is stored in a decentralized database, and the second decentralized database is... Figure 6 The reported data is stored in a decentralized database. After the source data completes the Map operation, the data to be processed is temporarily stored in the source data decentralized storage database, and the data to be processed is stored on adjacent nodes ( Figure 6 The crossed lines in the diagram indicate storage on adjacent nodes. After the data to be processed undergoes the Reduce operation, the outgoing data is obtained and stored in the decentralized storage database for the outgoing data.

[0122] In the embodiments disclosed herein, the decentralized storage technology IPFS is integrated with the traditional big data technology Hadoop to provide MapReduce operations and data processing for data on IPFS nodes nearby. This avoids the complex processes of data retrieval, transmission, and computation required by traditional centralized data lakes on global and cross-rack nodes, thereby reducing the communication costs between nodes and ultimately improving computing efficiency.

[0123] Next, taking the implementation scenario of bank regulatory reporting as an example, we will reveal in detail the data processing method based on blockchain in this scenario.

[0124] Figure 7 The diagram illustrates a complete flowchart of a blockchain-based data processing method according to an embodiment of the present disclosure.

[0125] like Figure 7 As shown, a blockchain-based data processing method in this embodiment includes operations S701 to S713. Operations S701 to S713 are respectively executed in the business application, the reporting blockchain, the first decentralized database, the second decentralized database, and the external reporting application.

[0126] In S701 operation, business operations occur in the business application, and business data is obtained.

[0127] Specifically, business applications generate relevant business data as business activities occur, and this data is also the source of data reported by regulators.

[0128] In operation S702, the business data is written into the business database.

[0129] The business database is a database that supports business applications and can be viewed as a small, centralized database.

[0130] In operation S703, the source data lake entry smart contract deployed on the blockchain is executed periodically to enable the source data lake entry logic.

[0131] In operation S704, the smart contract that sends source data from the blockchain to the lake writes the source data to the first decentralized database.

[0132] The first decentralized database stores only source data.

[0133] In the S705 operation, the lake entry status is recorded.

[0134] The entry status includes success (SUCCESS) and failure (FAILED).

[0135] In operation S706, the blockchain is reported to monitor the entry status into the lake. If successful, operation S707 is executed; if it fails, operation S713 is executed.

[0136] In the S707 operation, the smart contract for processing regulatory reporting data is executed to process the source data and obtain the reporting data.

[0137] During the operation of S708, data is reported into the lake and sent to the second centralized database.

[0138] In the S709 operation, the lake entry status is recorded.

[0139] In operation S710, the blockchain is reported to monitor the entry status into the lake. If successful, operation S711 is executed; if it fails, operation S713 is executed.

[0140] In the S711 operation, a smart contract is executed to generate regulatory reporting data files, which are then sent to external reporting applications.

[0141] In S712 operation, data files are received and reported externally.

[0142] During S713 operation, the lake entry failure status is sent to the external reporting application to trigger an alarm.

[0143] This invention proposes a regulatory reporting data stream system based on blockchain smart contracts. Through this invention, a decentralized reporting data processing network can be built within the runtime services of various reporting applications. Smart contracts handle the cleaning and processing logic of regulatory reporting data, generating regulatory indicator data and detailed data. This data is then aggregated in a decentralized storage database in real time through data streaming processing. Multiple reporting smart contracts then connect to external reporting applications in a mutually redundant manner. These smart contracts only implement data streaming processing, transmitting the regulatory reporting data from the decentralized storage database to the external reporting applications, where a summary data file is generated.

[0144] Based on the aforementioned blockchain-based data processing method, this disclosure also provides a blockchain-based data processing device. The following will be combined with... Figure 8 The device is described in detail.

[0145] Figure 8 A schematic block diagram of a blockchain-based data processing apparatus according to an embodiment of the present disclosure is shown.

[0146] like Figure 8 As shown, the blockchain-based data processing device 800 of this embodiment includes a data acquisition module 810, a data processing module 820, a file generation module 830, and a file transfer module 840.

[0147] The data acquisition module 810 is used to execute the lake entry smart contract based on the business blockchain, perform a write operation on the collected source data, and write it to the decentralized database. In one embodiment, the data acquisition module 810 can be used to execute the operation S210 described above, which will not be repeated here.

[0148] The data processing module 820 is used to execute a data processing smart contract based on the business blockchain, process the source data in the decentralized database to form outgoing data, and perform the write operation on the outgoing data to write it to the decentralized database. In one embodiment, the data processing module 820 can be used to execute the operation S220 described above, which will not be repeated here.

[0149] The file generation module 830 is used to execute a file generation smart contract based on the business blockchain to generate an outgoing file. In one embodiment, the file generation module 830 can be used to execute the operation S230 described above, which will not be repeated here.

[0150] The file transfer module 840 is used to transfer the external file to the target terminal. In one embodiment, the file transfer module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0151] In the embodiments disclosed herein, data collection and transmission are conducted entirely using blockchain in conjunction with a decentralized database, which greatly ensures data security and avoids the possibility of data tampering during the process. Using blockchain in conjunction with a decentralized database also effectively avoids the inefficiency of relying solely on blockchain for data storage and processing, thus improving the efficiency of the entire data processing workflow.

[0152] According to an embodiment of this disclosure, the source data and the outgoing data are inbound data, and the write operation includes: separating the inbound data into a first preset number of sub-data blocks; and using erasure coding to encode the first preset number of data blocks into a second preset number of data blocks, wherein the second preset number of data blocks has a third preset number of redundant data blocks compared to the first preset number of sub-data blocks.

[0153] According to an embodiment of this disclosure, after the first preset number of data blocks are written into a second preset number of data blocks using erasure coding, the write operation further includes: identifying the content in the sub-data blocks based on multiple hashes; and storing the identified sub-data blocks into a decentralized node based on a distributed hash table.

[0154] According to an embodiment of this disclosure, the data acquisition module 810 is further configured to periodically execute the lake entry smart contract based on a preset acquisition time.

[0155] According to embodiments of this disclosure, the decentralized database includes a first decentralized database and a second decentralized database, wherein the source data is written to the first decentralized database and the outgoing data is written to the second decentralized database.

[0156] According to embodiments of this disclosure, the data processing module 820 is further configured to perform a mapping operation on the source data in the first decentralized node to obtain data to be processed, wherein the first decentralized node is a node in the first decentralized database, and the mapping operation is used to rearrange data blocks; temporarily store the data to be processed in a second decentralized node, wherein the second decentralized node is a node in the first decentralized database and is an adjacent node of the first decentralized node; perform a simplification operation on the data to be processed on the second decentralized node based on preset calculation logic to obtain the outgoing data; and store the outgoing data in a third decentralized node, wherein the third decentralized node is a node in the second decentralized database.

[0157] According to an embodiment of this disclosure, the device further includes a monitoring and alarm module, wherein the monitoring module is used to monitor the warehousing status of the source data and / or monitor the warehousing status of the outgoing data; if the warehousing status is successful, a warehousing success command is issued; and if the warehousing status is unsuccessful, a warehousing failure command is issued to complete the alarm.

[0158] According to embodiments of this disclosure, any plurality of modules among the data acquisition module 810, the data processing module 820, the file generation module 830, and the file transfer module 840 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the data acquisition module 810, the data processing module 820, the file generation module 830, and the file transfer module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 810, the data processing module 820, the file generation module 830, and the file transfer module 840 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0159] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a blockchain-based data processing method according to an embodiment of the present disclosure.

[0160] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0161] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0162] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0163] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0164] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include one or more memories other than ROM 902 and / or RAM 903 described above.

[0165] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the blockchain-based data processing method provided in the embodiments of this disclosure.

[0166] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0167] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0168] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0169] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0171] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0172] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method based on blockchain, characterized in that, include: Based on the business blockchain, the smart contract for entering the lake is executed to perform a write operation on the collected source data and write it to a decentralized database. The decentralized database does not have a centralized management node. The decentralized database does not perform consensus and chain storage when storing data. Based on the business blockchain, a data processing smart contract is executed to process the source data in the decentralized database, forming outgoing data, and the write operation is performed on the outgoing data to write it into the decentralized database; and Based on the aforementioned business blockchain execution file, a smart contract is generated, and an external file is generated; and The exported file is transmitted to the target device; The decentralized database includes a first decentralized database and a second decentralized database, wherein the source data is written to the first decentralized database and the outgoing data is written to the second decentralized database.

2. The method according to claim 1, characterized in that, in, The source data and the externally transmitted data are the data to be stored in the database. The write operation includes: The data to be stored is divided into a first preset number of sub-data blocks; and Erasure coding is used to compile the first preset number of data blocks into a second preset number of data blocks, wherein the second preset number of data blocks contains a third preset number of redundant data blocks compared to the first preset number of sub-data blocks.

3. The method according to claim 2, characterized in that, After the first preset number of data blocks are encoded into a second preset number of data blocks using erasure coding, the write operation further includes: The content of the sub-data block is identified based on multiple hash identifiers; and The identified sub-data blocks are stored in decentralized nodes based on a distributed hash table.

4. The method according to claim 1, characterized in that, The execution of the smart contract for entering the lake based on the business blockchain includes: The smart contract for entering the lake is executed periodically based on a preset collection time.

5. The method according to claim 4, characterized in that, The step of executing a data processing smart contract based on the business blockchain to process the source data in the decentralized database to form outgoing data, and storing the outgoing data in the decentralized database, includes: A mapping operation is performed on the source data in the first decentralized node to obtain data to be processed, wherein the first decentralized node is a node in the first decentralized database, and the mapping operation is used to rearrange data blocks; The data to be processed is temporarily stored in a second decentralized node, wherein the second decentralized node is a node in the first decentralized database and is an adjacent node of the first decentralized node; Based on preset computational logic, the data to be processed on the second decentralized node is simplified to obtain the outgoing data; and The outgoing data is stored in a third decentralized node, wherein the third decentralized node is a node in the second decentralized database.

6. The method according to claim 1, characterized in that, After the source data is written to the decentralized database, or after the outgoing data is written to the decentralized database, the method further includes: monitoring the entry status of the source data and / or monitoring the entry status of the outgoing data; If the warehousing status is successful, then a warehousing success command is issued; and If the inbound status is failed, an inbound failure command is issued to complete the alarm.

7. A data processing device based on blockchain, characterized in that, The device includes: a data acquisition module, a data processing module, a file generation module, and a file transfer module. in, The data acquisition module is used to execute the lake entry smart contract based on the business blockchain, perform a write operation on the collected source data, and write it to a decentralized database, wherein the decentralized database does not have a centralized management node; wherein the decentralized database does not perform consensus and chain storage when storing data; The data processing module is used to execute a data processing smart contract based on the business blockchain, process the source data in the decentralized database to form outgoing data, and perform the write operation on the outgoing data to write it into the decentralized database. The file generation module is used to execute a file generation smart contract based on the business blockchain to generate an external file; and The file transfer module is used to transfer the external file to the target terminal; The decentralized database includes a first decentralized database and a second decentralized database, wherein the source data is written to the first decentralized database and the outgoing data is written to the second decentralized database.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images