A data collection and processing method and system for a blockchain

Abstract

The present application relates to a kind of data acquisition and processing method for blockchain, new block information received by blockchain node is collected according to data acquisition scale by data acquisition data structure;New block information is processed according to data processing data model matched with the data acquisition data structure;A kind of data acquisition and processing system for blockchain is used to realize the above method, including data aggregation agent module and multiple data reporting service modules;The data reporting service module obtains new block information and sends new block information to data aggregation agent module;The data aggregation agent module carries out data aggregation and processing to the new block information received.This application completes the data acquisition and processing of blockchain node according to data acquisition scale by constructing data acquisition data structure and data processing data model, and the data result obtained is good in universality, realizes the unified standard of data result, and is convenient for subsequent use.

Description

Technical Field

[0001] This invention relates to the field of blockchain, and in particular to a method and system for data acquisition and processing in blockchain. Background Technology

[0002] Blockchain systems contain a wide variety of publicly available historical and real-time data. Collecting and analyzing this data helps to identify and summarize patterns and anomalies within the blockchain system. In the current blockchain ecosystem, there are four main types of target data sources: 1) Blockchain system components, including address information, transaction and block information maintained by each component; 2) Transaction entities, primarily referring to verified account and address information; 3) Peer nodes, including node operational information and network behavior; and 4) Various external data sources, including financial indices and search trends.

[0003] For the aforementioned data sources, the mainstream data collection methods currently include: 1) crawling public datasets and blockchain explorers using web crawlers to collect block and transaction data; 2) directly parsing the blockchain copies stored by peer nodes to extract block and transaction data; 3) independently deploying nodes and collecting public data from other peer nodes by using RPC calls and protocol information exchange based on peer-to-peer network communication protocols; and 4) using instrumentation techniques to intercept or read the internal runtime data of peer node clients.

[0004] The aforementioned data collection methods all have some drawbacks. Web scraping and parsing blockchain copies can only collect deterministic block and transaction data, failing to obtain real-time data such as orphan blocks, stale blocks, orphan transactions, and transaction pool data. Data collection based on peer-to-peer communication protocols has significant limitations. RPC calls require verification information from other peer nodes, which is insecure and impractical in a large peer-to-peer network. Data collected through protocol information exchange depends entirely on the target peer node, making the data collection process difficult to predict and control. Furthermore, both RPC calls and protocol information exchange yield relatively limited data richness, insufficient for most usage scenarios. Instrumentation techniques require custom modifications to the client running on the peer node to complete the aforementioned data collection. This presents significant challenges, such as defining data standards, handling heterogeneous data, and ensuring data collection independence. It also only provides access to internal data within the peer node client, such as contract execution data, and cannot collect external data. Therefore, in most cases, it is necessary to combine several data collection methods, which adds considerable difficulty and workload to the data collection work.

[0005] Some researchers utilize the RPC protocol of blockchain systems to obtain real-world data. For example, Kalodner and his colleagues used the RPC protocol to acquire and analyze transaction data within blockchain systems. However, using the RPC protocol for information acquisition has drawbacks such as high overhead and limited information volume. Other researchers obtain real-world data by analyzing existing system logs. For instance, Zheng used regular expressions to match log files and obtain data, studying how to measure the performance of blockchain systems using real-world data. However, the original logs of blockchain systems contain very little information. Matching log files using regular expressions can only collect specific information, limiting the widespread use of their data. Some researchers observe node-to-blockchain data by instrumenting nodes within Ethereum's code. For example, Chen et al. collected internal and external transaction data, smart contract creation and invocation data, and transaction record data of Ethereum nodes by instrumenting them into full Ethereum nodes. Their research focuses more on Ethereum's smart contracts and cannot be migrated to the blockchain system.

[0006] The aforementioned traditional technologies have the following drawbacks in terms of technical effectiveness:

[0007] 1. Web scraping technology and direct parsing of blockchain copies. This method can only collect deterministic block and transaction data, not real-time data.

[0008] 2. Data collection technology based on peer-to-peer network communication protocols. Insecure or impractical, cannot be applied to real-world peer-to-peer networks; insufficient information richness, significant limitations; high overhead, small data volume, poor collection efficiency; affects the independence of the collection process from the blockchain system; the collection process is unpredictable and uncontrollable.

[0009] 3. Data is parsed using regular expressions based on code instrumentation techniques. However, this method is rudimentary and cannot be easily migrated; the collected data results have poor general applicability and lack a unified standard; and for heterogeneous data, the data extraction workload is enormous.

[0010] Currently, there is considerable data collection work targeting various types of data. However, the collected data lacks mature data collection solutions and there is a lack of work specifically addressing data related to the blockchain formation process. Moreover, existing data collection systems are more about developing data applications based on existing data collection solutions, rather than deeply implementing and deploying these solutions. They fail to consider or address the various challenges that may arise during the implementation of data collection solutions, and they do not provide supporting data sharing mechanisms. This results in significant waste of collected data, making it difficult to effectively acquire and utilize. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data collection and processing method and system for blockchain, which yields data results with good versatility, achieves a unified standard for data results, and facilitates subsequent use.

[0012] The objective of this invention is achieved through the following technical solution:

[0013] A data collection and processing method for blockchain includes the following steps:

[0014] S1: Collect new block information received by blockchain nodes based on the data collection data structure and data collection scale;

[0015] S2: Process the new block information according to the data processing data model that matches the data acquisition data structure;

[0016] If the blockchain node is a single node, then the data collection scale includes the block synchronization process, the block synchronization stage, and the block synchronization event.

[0017] If the blockchain node consists of multiple nodes, the data collection scale includes time, block synchronization process, block synchronization stage, and block synchronization event.

[0018] Furthermore, the data acquisition data structure includes a block synchronization process snapshot, a block synchronization stage revision, a block synchronization event, and the blockchain system state BestState;

[0019] The block synchronization process Snapshot packages the synchronization process data in the new block information;

[0020] The block synchronization phase information is included in the block synchronization phase information packaged in the Revision phase of the block synchronization phase.

[0021] The block synchronization event is packaged into the new block information.

[0022] The system running status information in the BestState package of the blockchain system state;

[0023] The data processing data models include: Snapshot data model, Revision data model, Event data model, and BestState data model.

[0024] Furthermore, the block synchronization phase Revision includes InitData and FinalData fields. InitData can be any custom data type, and FinalData can be any custom data type.

[0025] The block synchronization event information refers to block synchronization event information related to orphan blocks.

[0026] Furthermore, the collection of new block information in step S1 is achieved through a data instrumentation method;

[0027] The data instrumentation method uses the TraceData data structure to collect new block information;

[0028] The TraceData data structure includes InitSnapshot and FinalSnapshot fields.

[0029] Furthermore, the instrumentation output of the data instrumentation method adopts a universal interface.

[0030] A data acquisition and processing system for blockchain, used to implement the aforementioned data acquisition and processing method for blockchain, includes a data aggregation agent module and multiple data reporting service modules;

[0031] The data reporting service module obtains new block information by reading and listening to the log output of blockchain nodes and sends the new block information to the data aggregation agent module;

[0032] The data aggregation proxy module performs data aggregation and processing on the received new block information.

[0033] Furthermore, the data reporting service module uses a passive listening method to read and monitor the log output of the blockchain nodes.

[0034] Furthermore, the data aggregation proxy module includes a receiving submodule and multiple data processing service submodules, and the data processing service submodules correspond one-to-one with the data reporting service module;

[0035] The receiving submodule distributes the new block information to the corresponding data processing service submodule according to the information source of the new block information.

[0036] Furthermore, the receiving submodule includes a Receiver Server, an MQ Server, and a Meta Server;

[0037] The Receiver Server receives new block information from the data reporting service module and sends the new block information to the MQ Server and Meta Server. After receiving the new block information, the MQ Server communicates with the Meta Server to confirm the real-time performance and integrity of the data. After confirming that there are no errors, the MQ Server transmits the new block information to the corresponding data processing service sub-module according to the corresponding information between the data reporting service module and the data processing service sub-module. After receiving the new block information, the Meta Server determines whether packet loss has occurred and decides whether to notify the data reporting service module to retransmit.

[0038] Furthermore, the data processing service submodule processes the new block information according to the data processing data model;

[0039] The data processing service submodule stores the processed data obtained from processing the new block information in the data storage service module.

[0040] The data storage service module is equipped with a data access interface.

[0041] The beneficial effects of this invention are:

[0042] This invention constructs a data acquisition data structure and a data processing data model, and completes the data acquisition and processing of blockchain nodes according to the data acquisition scale. The obtained data results have good versatility, realize a unified standard for data results, and are convenient for subsequent use. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a data acquisition and processing system for blockchain.

[0044] Figure 2 This is a structural diagram of a data acquisition and processing system for blockchain.

[0045] Figure 3 This is a schematic diagram of the receiving submodule;

[0046] Figure 4 This is a structural diagram of MsgServer and Resolver. Detailed Implementation

[0047] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0048] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0049] Example 1:

[0050] like Figures 1 to 4 As shown, a data collection and processing method for blockchain is presented.

[0051] Includes the following steps:

[0052] S1: Collect new block information received by blockchain nodes based on the data collection data structure and data collection scale;

[0053] S2: Process the new block information according to the data processing data model that matches the data acquisition data structure;

[0054] If the blockchain node is a single node, then the data collection scale includes the block synchronization process, the block synchronization stage, and the block synchronization event.

[0055] If the blockchain node consists of multiple nodes, the data collection scale includes time, block synchronization process, block synchronization stage, and block synchronization event.

[0056] Description of the blockchain formation process under a single node:

[0057] In a blockchain, each fully functional peer node runs an independent client and maintains a complete copy of the blockchain. Through multiple block synchronization processes, it reaches consensus with the majority of peer nodes in the network regarding the blockchain copy. Each peer node simultaneously establishes peer relationships with multiple other peer nodes, concurrently performing multiple block synchronization processes to ultimately form its local blockchain copy. From the perspective of a single peer node, the formation of the blockchain is a process of multiple block synchronizations among peer nodes. Therefore, we can describe the concurrent block synchronization process to achieve an equivalent description of the blockchain formation process.

[0058] From a single-node perspective, the formation process of a blockchain can be described at three scales: concurrent block synchronization, sequential block synchronization phases, and dynamic block synchronization events. Each independent peer node concurrently performs multiple block synchronizations, and each block synchronization process goes through multiple sequential block synchronization phases. During each block synchronization phase, unpredictable block synchronization events occur randomly, such as blocks being added to or removed from the chain, or orphan blocks being created and discarded. As the descriptive scale deepens, we can gradually observe increasingly detailed micro-data.

[0059] This section first introduces the concept of BestState. In a blockchain system, the BestState describes the current best block and various information related to the best chain, such as the best chain height, network difficulty, total number of transactions, and median time. Typically, changes to BestState can be used as a benchmark to describe changes in the local blockchain. Each block synchronization process may result in multiple changes to BestState. Therefore, at the descriptive scale of the block synchronization process, we can describe these concurrent block synchronization processes based on the start and end timestamps, the start and end BestState, and the associated block hash information. Similarly, when refining the description to the block synchronization stage, each block synchronization stage may also cause changes to BestState. Therefore, we can describe each block synchronization stage based on its start and end timestamps and the key information at the start and end. This key information has different definitions depending on the current synchronization stage.

[0060] Describing the block synchronization process and block synchronization phase both describe a time interval within the blockchain system's operation. Therefore, at these two scales, we only need to focus on the start and end information of the time block. Describing block synchronization events, however, describes a specific point in time within the blockchain system's operation. Block synchronization events represent actions that nodes may take at any time during the block synchronization phase, and the results of those actions. By comprehensively describing block synchronization events, we can present the actions and results of peer nodes at every moment in the most detailed way possible, but this also increases data redundancy and brings a lot of unnecessary work to data processing and use. Therefore, when describing events, we must determine which events to describe and which to ignore based on the requirements. This embodiment focuses on describing events related to orphan blocks during the block synchronization phase. The descriptive information mainly includes the timestamp information of the event, the event type, and the hash of the orphan block associated with the event.

[0061] Description of the blockchain formation process under multiple nodes:

[0062] In a multi-node scenario, each node competes for the right to record transactions next, following the Proof-of-Work (PoW) consensus mechanism. The peer node that wins this right connects its local blockchain to the peer network and broadcasts the block. Other peer nodes in the network maintain consistency between their local blockchain copy and the majority of nodes by performing multiple block synchronization processes. Among multiple peer nodes, interactions are limited to necessary activities such as block synchronization, block propagation, and peer discovery. Each peer node's block synchronization process, its specific stage, and the events that occur are independent and do not affect each other. Therefore, from the perspective of multiple peer nodes, the formation of a blockchain is a process of multiple block synchronizations performed by multiple peer nodes; the block synchronization process is the basic unit describing the blockchain formation process.

[0063] Therefore, similar to the single-node perspective, the blockchain formation process from a multi-node perspective can also be described using three scales: block synchronization process, block synchronization phase, and block synchronization event. The slight difference is that a time scale needs to be introduced in the multi-node perspective. Although the synchronization processes of multiple nodes are independent, according to the consensus protocol, these nodes will eventually reach a unified consensus on their local blockchain copies. In other words, these independent synchronization processes on these nodes need to be merged to fully describe the blockchain formation process in a multi-node context. At the time-based descriptive scale, using the block synchronization process as the basic unit, reorganizing the multiple independent synchronization processes on multiple nodes according to their chronological order completes the description of the blockchain formation process on the time scale.

[0064] Based on the above description of the blockchain formation process, a data collection structure was developed to represent the blockchain formation process data at different data collection scales.

[0065] The data acquisition data structure includes a block synchronization process snapshot, a block synchronization stage revision, a block synchronization event event, and the blockchain system state BestState.

[0066] The block synchronization process Snapshot packages the synchronization process data in the new block information;

[0067] Block synchronization process Snapshot. The Snapshot structure, literally meaning a snapshot, packages all data from the entire block synchronization process into a snapshot, representing the current state of the blockchain system. The Snapshot structure is used to describe the blockchain formation process at the synchronization scale. The data in the block synchronization process Snapshot includes, but is not limited to, the timestamps of the start and end of the block synchronization process, as well as the blockchain system state information at the start and end. The block synchronization process Snapshot structure includes the following data fields:

[0068] SnapshotID. SnapshotID is a unique identifier for the current block synchronization process. The ID consists of three parts: the height of the main chain or side chain where the current block resides (forkheight = 0 indicates the current block is on the main chain), the height of the current block (chainheight), and the timestamp indicating the start of the current block synchronization process. Since the parent block of the current block may not exist in the local blockchain, if the parent block can be found, forkheight can be obtained normally; if the parent block cannot be found, forkheight is set to -1, indicating that the current block is an orphan block.

[0069] SnapshotType. SnapshotType represents the type of the current snapshot. There are two main types: InitSnapshot and FinalSnapshot, which represent data snapshots at the start and end times of the current block synchronization process, respectively. The two types of snapshots are combined to represent a block synchronization process.

[0070] Timestamp. The timestamp information of the current snapshot, accurate to nanoseconds.

[0071] BestState. The current state of the system at the point in time of the current snapshot.

[0072] BlockHash. The hash information of the block corresponding to the current block synchronization process.

[0073] IsOrphan. Whether the block corresponding to the current block synchronization process is an orphan block, that is, whether the block becomes an orphan block after the current block synchronization process.

[0074] RevisionList. The current block synchronization process includes multiple sequential block synchronization stages.

[0075] EventList. Multiple events occur in chronological order during the current block synchronization process.

[0076] The block synchronization phase information is included in the block synchronization phase information packaged in the Revision phase of the block synchronization phase.

[0077] Block synchronization phase Revision. The literal meaning of Revision is revision; each Revision structure represents a block synchronization phase that may alter the current system state. The Revision structure describes the blockchain formation process at the scale of the synchronization phase. The data for a block synchronization phase Revision includes, but is not limited to, the timestamps of the start and end of the phase, and important phase information at the start and end. The block synchronization phase Revision structure includes the following data fields:

[0078] SnapshotID. The ID of the block synchronization process to which the current block synchronization phase belongs.

[0079] RevisionType. The type of the current block synchronization stage, determined according to the specific block synchronization process.

[0080] InitTimestamp. The timestamp indicating the start of the current block synchronization phase.

[0081] InitData. Key information relevant at the start time of the current block synchronization phase. To ensure sufficient data scalability, InitData can be any custom data type.

[0082] FinalTimestamp. The timestamp indicating the end of the current block synchronization phase.

[0083] FinalData. Similar to InitData, it represents the key information relevant at the end of the current block synchronization phase. To ensure sufficient data scalability, FinalData can be any custom data type.

[0084] The block synchronization event is packaged into the new block information.

[0085] The block synchronization event information refers to block synchronization event information related to orphan blocks;

[0086] Block synchronization events (Events). Considering the dynamic, unpredictable, and diverse nature of block synchronization events, this section focuses only on block synchronization events related to orphan blocks. The data for an Event includes, but is not limited to, the timestamp of the event, the event type, and the hash of the orphan block associated with the event. EventOrphan includes the following data fields:

[0087] SnapshotID. The ID of the block synchronization process to which the current event belongs.

[0088] EventType. Event type. For orphan block related events, the types include: orphan block creation event, orphan block on-chain event, and orphan block discard event.

[0089] OrphanParentBlockHash. The hash of the parent block of the current orphan block.

[0090] OrphanBlockHash. The hash of the current orphan block. An orphan block typically goes through a process from its creation to its addition to the blockchain or its discard. Therefore, based on these two hashes, the entire orphan block event flow can be traced.

[0091] ConnectMainChain. When the orphan block event type is "orphan block on-chain", this field is used to distinguish whether the orphan block has been added to the main chain or the side chain.

[0092] Timestamp. The timestamp information of when the event occurred.

[0093] The system running status information in the BestState package of the blockchain system state;

[0094] The blockchain system state is BestState. The BestState structure contains a series of information about the system's operating state; the specific data fields will not be described in detail here due to space limitations.

[0095] The data collection structure described above completes the description of the blockchain formation process under a single node. Under multiple nodes, based on the timestamp information provided in the collected data, it is easy to complete the description of the blockchain formation process at the data level over a time scale.

[0096] Corresponding to the data acquisition data structure, the data processing data model includes: Snapshot data model, Revision data model, Event data model, and BestState data model.

[0097] Since the raw data has already been well parsed and organized during the data acquisition, data structure, and instrumentation output stages, the specific fields of the data processing data model are similar to the data structure design. In addition to the design of the data model fields, the settings for the primary keys and indexes in the data model are shown in the description tables for each data model. Yellow keys represent primary keys, multiple yellow keys represent composite primary keys, and green keys represent the set indexes.

[0098] Description of the Snapshot data model:

[0099]

[0100] In the table, the key for snapshot_id is yellow, and the keys for target_chain_id, target_chain_height, and block_hash are green.

[0101] The Snapshot model combines InitSnapshot and FinalSnapshot data types, representing a complete block synchronization process. The SnapshotID is also stored as a foreign key in other data models for easy traceability and retrieval. Additionally, the Snapshot model includes the following indexes:

[0102] 1) A composite index of chainid and chainheight. This index allows you to query the synchronization process data of blocks at any height on a given chain (including the main chain). In extreme cases, multiple blocks may have the same chainid and chainheight but be located on different sidechains; in such cases, this index may return multiple query results.

[0103] 2) Parentblockhash, a composite index of chainid. Compared to the previous index, this index has a larger index size, but it can uniquely locate any block on a chain (including the main chain) and query the synchronization process data of that block.

[0104] 3) Blockhash Index. This index provides the ability to directly query the block synchronization process corresponding to a block through its hash. The blockhash index is designed to handle situations where the chainid and chainheight cannot be obtained when a block becomes an orphan during block synchronization. Based on the blockhash index, when an orphan block is added to the chain or discarded during subsequent block synchronization, the block synchronization process that generated the orphan block can be located through the hash of the orphan block in the event.

[0105] Description of the Revision data model:

[0106]

[0107] In the table, the keys for snapshot_id, revision_type, and init_timestamp are highlighted in yellow.

[0108] The Revision data model is linked to the Snapshot model through the SnapshotID foreign key. Revision uses three fields, SnapshotID, RevisionType, and InitTimestamp, as a composite primary key. The reason is that a block synchronization process goes through multiple block synchronization stages, and the same block synchronization stage may be performed recursively. Stages such as OrphanProcess may occur repeatedly, and SnapshotID and RevisionType alone cannot uniquely represent a Revision.

[0109] Description of the Event data model:

[0110]

[0111] In the table, the keys for snapshot_id and event_type_orphan are yellow, and the key for orphan_bloch_hash is green.

[0112] Description of the BestState data model:

[0113]

[0114] In the table, the keys for snapshot_id and snapshot_type are highlighted in yellow.

[0115] Similar to the Revision data model, the primary key for the Event and BestState data models is a composite primary key of SnapshotID and the corresponding type information. The Event model additionally adds indexes to OrphanParentBlockHash and OrphanBlockHash, supporting queries for related orphan blocks by parent block hash and queries for related orphan block events by block hash.

[0116] By designing the data acquisition data structure and the data processing data model, the data standard was established.

[0117] The collection of new block information in step S1 is achieved through data instrumentation.

[0118] After step S1 (i.e., the raw data is collected and output to the outside through instrumentation) is completed, step S2 is needed to complete the data processing stage in order to process the raw data according to the definition of the data processing data model and store it in the database for better use later.

[0119] Design and implementation of data acquisition scheme

[0120] Design of data acquisition scheme

[0121] Data acquisition technology selection. Since the data acquired in this embodiment consists of various micro-data generated in real time at peer nodes, the requirements for data real-time performance and richness are very high. Therefore, in order to realize all the data acquisition functions of this system, this embodiment will adopt data instrumentation technology as the basis for data acquisition.

[0122] Design principles. The following design principles should be followed when designing a data acquisition scheme:

[0123] Independence of data acquisition. Since the data acquisition method of the data instrumentation code and the output method of the instrumentation data can affect the host program, leading to incorrect acquisition results or even affecting the normal operation of peer node clients, it is necessary to optimize the independence of the data acquisition process from both the writing of instrumentation code and the output of instrumentation data.

[0124] Meets data standards. To cope with various complex and heterogeneous data, after extracting the raw data, it is necessary to perform certain preprocessing work to ensure that the data output by instrumentation meets the established data standards, thereby supporting the subsequent management of data integrity, validity, consistency, standardization, openness and sharing.

[0125] Data instrumentation scalability. Due to the large workload and high complexity of data instrumentation, to adapt to ever-changing data needs, data instrumentation must provide a certain degree of scalability in both data acquisition and output. This allows for easy modification of acquired data while fully meeting existing data standards, without requiring extensive code changes. Simultaneously, instrumented data can be output through various alternative methods, achieving separation between data instrumentation code and output methods.

[0126] Implementation of data acquisition scheme

[0127] Based on the above design principles, the data acquisition scheme implemented in this embodiment is detailed as follows:

[0128] The data instrumentation method uses the TraceData data structure to collect new block information;

[0129] The TraceData data structure is used to organize all the data for a single block synchronization process at runtime;

[0130] The TraceData data structure includes InitSnapshot and FinalSnapshot fields.

[0131] As mentioned earlier, the InitSnapshot and FinalSnapshot fields represent data snapshots at the start and end points of the block synchronization process, respectively. Since a block synchronization process in the code is represented by a series of sequential function calls, the main purpose of the TraceData data structure is to reuse collected data and organize newly added data through parameter passing between these function calls. Finally, by instrumenting data at appropriate locations in each function, the extracted data is stored in TraceData according to data standard requirements.

[0132] Data instrumentation extension. Regarding data instrumentation code, firstly, this solution provides a rich set of instrumentation functions, allowing for easy addition of new data instrumentations at any location. Secondly, as mentioned earlier, the InitData and FinalData in each Revision structure can be any custom data type, thus ensuring the diversity of data collection. Other personnel can easily add the data to be collected at any location, only needing to complete the corresponding data extraction code.

[0133] Instrumentation Data Output. Regarding instrumentation output, this solution defines a general interface for instrumentation output. The instrumentation output method can be replaced by any method that implements the interface, such as HTTP, log files, RCP, etc. Considering the independence of data acquisition, this solution initially provides an interface for outputting raw data via logs. This is because network bandwidth and various computing resources are factors that affect the operation of peer nodes, while the log output method only involves physical disk I / O.

[0134] The data instrumentation method uses a universal interface for its instrumentation output.

[0135] The general interfaces include, but are not limited to, HTTP, log files, and RCP.

[0136] The above solution successfully collected all the necessary data at the peer nodes, and the collected data met the data standard requirements. Simultaneously, the entire data acquisition process maintained independence from the operation of the peer nodes and provided sufficient scalability.

[0137] A data acquisition and processing system for blockchain, used to implement the aforementioned data acquisition and processing method for blockchain, includes a data aggregation agent module and multiple data reporting service modules;

[0138] The data reporting service module obtains new block information by reading and listening to the log output of blockchain nodes and sends the new block information to the data aggregation agent module;

[0139] The data aggregation proxy module performs data aggregation and processing on the received new block information.

[0140] The data reporting service module and the data aggregation agent module are electrically connected.

[0141] The data reporting service module, also known as the Exporter, underwent a process where two methods were compared to ensure real-time and lossless collection of node log data. The first method involved the Exporter actively polling nodes to obtain relevant data. The second method involved the Exporter passively listening to nodes to obtain relevant data. The active polling method struggled to guarantee real-time data acquisition. Specifically, short polling times consumed node network resources, leading to data loss. Long polling times consumed node memory resources, causing memory overflow and data loss. Therefore, we chose the passive listening method, which consumes fewer resources and provides real-time data feedback. It's important to note that the log data obtained through passive listening is not a sequential node synchronization process and still requires sorting by subsequent components.

[0142] The data reporting service module uses a passive listening method to read and monitor the log output of the blockchain nodes.

[0143] After the Exporter collects relevant data, it sends the data to the Receiver, a data aggregation proxy module deployed outside the network, via the HTTP protocol for subsequent data aggregation.

[0144] The data aggregation proxy module includes a receiving submodule and multiple data processing service submodules, and the data processing service submodules correspond one-to-one with the data reporting service module.

[0145] The data reporting service module and the data aggregation agent module are electrically connected to the receiving submodule; the data processing service submodule and the receiving submodule are electrically connected.

[0146] The data aggregation agent module, also known as the Agent.

[0147] The data aggregation function is designed to facilitate the system's parallel processing of metadata from multiple different nodes, and to perform data management and distribution. The complete functionality of the Agent is accomplished through the collaboration of the Receiver data aggregation service, the Meta metadata storage service, and the Mq message queue service; these three components together are referred to as the Agent.

[0148] In this system, this function is implemented by the receiving submodule (also known as Receiver).

[0149] The Receiver assigns a dedicated data processing service submodule to each Exporter based on the information source and saves the data transmission progress of the Exporter. This facilitates timely reminders to the Exporter to retransmit in case of packet loss due to network fluctuations, ensuring no data loss. Meanwhile, the Receiver itself does not process the data; instead, it delivers it to the data processing service submodule as quickly as possible to maintain data real-time performance.

[0150] The receiving submodule distributes the new block information to the corresponding data processing service submodule according to the information source of the new block information;

[0151] The receiving submodule includes a Receiver Server, an MQ Server, and a Meta Server;

[0152] The Receiver Server receives new block information from the data reporting service module and sends the new block information to the MQ Server and Meta Server. After receiving the new block information, the MQ Server communicates with the Meta Server to confirm the real-time performance and integrity of the data. After confirming that there are no errors, the MQ Server transmits the new block information to the corresponding data processing service sub-module according to the corresponding information between the data reporting service module and the data processing service sub-module. After receiving the new block information, the Meta Server determines whether packet loss has occurred and decides whether to notify the data reporting service module to retransmit.

[0153] In the receiving submodule, the Receiver Server communicates with the Exporter via HTTP and delivers the data to the MQ Server and Meta Server. Upon receiving the data, the MQ Server first filters it and communicates with the Meta Server to determine if there are any issues with the data's real-time performance and integrity. After confirmation, the MQ Server delivers the data to the corresponding data processing service submodule based on its stored Exporter-to-data-processing service submodule information. Upon receiving the data, the Meta Server checks its stored Exporter data progress to determine if packet loss has occurred and whether to instruct the Exporter to retransmit. Through the cooperation of the MQ Server and Meta Server, this module achieves data aggregation and distribution as quickly as possible without data loss, maintaining data real-time performance.

[0154] The data processing service submodule processes the new block information according to the data processing model.

[0155] The data processing service submodule implements data processing functions, with the aim of processing the real-time data collected by the system according to the data processing data model defined in the data standard.

[0156] The data processing service submodule includes a Resolver. A MsgServer is connected between the receiving submodule and the Resolver. The MsgServer filters and distributes data containing new block information, and the Resolver completes the data processing. To efficiently and concurrently process information from multiple Exporters, the Resolver only processes information that matches its own. Therefore, the MsgServer first filters information from the MQ Server based on the Exporter's information. Then, it sends the data to the corresponding Resolver. It's worth noting that during implementation, the data processing service submodule polls and processes data from the Receiver. This ensures data integrity and real-time performance. After acquiring data, the data processing service submodule processes and sorts the information according to real-time data as quickly as possible, resolving potential out-of-order issues during data transmission and ensuring the correctness of data processing.

[0157] The architecture of MsgServer and Resolver conforms to a typical distributed master-slave architecture in distributed computing. MsgServer handles management and task distribution, while Resolver performs the specific processing tasks; all Resolvers have the same functionality.

[0158] The data processing service submodule is electrically connected to the data storage service module.

[0159] The data processing service submodule stores the processed data obtained from processing the new block information in the data storage service module.

[0160] The data storage service module is also known as Collector.

[0161] The data processing service submodule will hand over the snapshot and timeline data to the Collector for storage.

[0162] The data storage service module implements data storage functionality. The Collector module is implemented using MySQL and deployed with a read-write separation architecture. The Master handles writes, and the Slave handles reads, optimizing data access efficiency to meet the demands of large-scale data processing.

[0163] The design and implementation of a data acquisition and processing system for blockchain fully adheres to the design principles of microservice architecture, including the single responsibility principle and the service autonomy principle. This decoupling of the system's functional modules enhances the dispersion of system business functions, achieves service autonomy through service-oriented deployment, and provides sufficient availability, scalability, and high elasticity.

[0164] The core components of a data acquisition and processing system for blockchain are Exporter, Agent, and Collector; its function is to collect, analyze, and store data on the blockchain formation process of client nodes.

[0165] The system supports deployment and collaboration across multiple client nodes for data completion. The general system logic is as follows: After a node receives new block information, the Exporter sends the received new block information (the block and its related information) to a Receiver deployed outside the network. Upon receiving the data, the Receiver distributes the new block information to its corresponding data processing service submodule for processing. After receiving the real-time data from the Receiver, the data processing service submodule processes the data according to the events based on the corresponding node's historical blockchain data and restores the node's blockchain state. After processing the data, the data processing service submodule packages the data and sends it to the Collector for storage.

[0166] The data storage service module is equipped with a data access interface.

[0167] The data access interface completes the data open service.

[0168] The data open service provides a series of data access interfaces, allowing external parties to access collected data by calling these interfaces. Furthermore, the data access interfaces also support multi-node data fusion at the data open level, using the fused data to reconstruct the formation process of the blockchain across multiple nodes. The scalability and compatibility of the data interfaces were fully considered during the design phase. By encapsulating the already stored data, the data open service provides more interfaces for convenient data access.

[0169] All fully open data access interfaces are accessed via the GET method, while interfaces accessible only to a limited number of users are accessed via the POST method, and access control information such as username, password, or access token must be provided. The unified format for the access interface links is as follows:

[0170] https: / / <interface address> / <interface version> / <interface type> / <specific interface>?<optional parameter 1>&<optional parameter 2>&...

[0171] Data access interfaces fall into three categories: query interfaces for the basic data model under a single node, query interfaces for multi-node data fusion, and data analysis and visualization interfaces. Among these, the multi-node data fusion and data analysis and visualization interfaces, due to their consumption of computing resources and potential impact on database operation, will only be selectively provided externally. The recommended approach is to use the basic data model query interface for independent processing and analysis.

[0172] This embodiment selects a blockchain system as the basic platform and implements a distributed data acquisition system for the blockchain formation process data. The core content of this embodiment mainly includes the following three parts:

[0173] 1. Design and Implementation of a Data Acquisition and Processing Solution for Blockchain. The blockchain formation process is described from both single-node and multi-node perspectives. Data standards are established to represent this process, encompassing both data structure and data model design. Finally, the overall solution is designed and implemented. Based on data instrumentation technology, a customized node client is used to output the raw data to be collected according to the established data standards through logs during code runtime.

[0174] 2. Design and Implementation of a Data Acquisition and Processing System for Blockchain. Based on the designed data acquisition and processing scheme, the data acquisition and processing process was optimized and implemented in terms of control capabilities, acquisition scale, and application capabilities. A microservice and distributed system architecture was adopted to complete the parallel acquisition and processing of massive amounts of data from multiple nodes and multiple data sources, and the data access efficiency under massive data was optimized. Full lifecycle control of multiple parallel acquisition processes was also achieved, and the application capabilities of the acquisition system were optimized in many aspects, including the system's real-time performance, reliability, correctness, scalability, elasticity, and availability.

[0175] 3. Design and Implementation of Open Data Services. Based on the developed data acquisition and processing system, open data services are provided at the data layer, allowing external users to easily access the collected data through data access interfaces. The open data services also provide interfaces for multi-node data fusion, data analysis, and visualization at the data layer, enabling the representation of the blockchain formation process across multiple nodes and better supporting data applications such as analysis and visualization.

[0176] A data acquisition and processing method and system for blockchain implements a real-time data collection and visualization tool based on peer nodes, providing three main functions: 1) It provides a data acquisition solution suitable for collecting data during the blockchain formation process. The solution is mature in design, highly scalable, and easy to implement; 2) It provides a readily usable, already implemented data acquisition system. The system supports parallel acquisition and processing of massive amounts of data from multiple nodes and has been optimized in terms of data storage, acquisition control capabilities, and system application capabilities; 3) Based on open data services, it provides various data access interfaces that facilitate data sharing and use, enabling the collected data to be utilized more conveniently and widely, maximizing the inherent value of the data.

[0177] By constructing a data acquisition data structure and a data processing data model, and based on the data acquisition scale, the data acquisition and processing of blockchain nodes were completed. The resulting data has good versatility, achieves a unified standard for data results, and is convenient for subsequent use.

[0178] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A data acquisition and processing method for blockchain, characterized in that: Includes the following steps: S1: Collect new block information received by blockchain nodes based on the data collection data structure and data collection scale; S2: Process the new block information according to the data processing data model that matches the data acquisition data structure; If the blockchain node is a single node, then the data collection scale includes the block synchronization process, the block synchronization stage, and the block synchronization event. If the blockchain node consists of multiple nodes, the data collection scale includes time, block synchronization process, block synchronization stage, and block synchronization event. The data acquisition data structure includes a block synchronization process Snapshot, a block synchronization phase Revision, a block synchronization event Event, and the blockchain system state BestState; The block synchronization process Snapshot packages the synchronization process data in the new block information; The block synchronization phase information is included in the block synchronization phase information packaged in the Revision phase of the block synchronization phase. The block synchronization event is packaged into the new block information. The system running status information in the BestState package of the blockchain system state; The data processing data models include: Snapshot data model, Revision data model, Event data model, and BestState data model.

2. The data acquisition and processing method for blockchain according to claim 1, characterized in that: The block synchronization phase Revision includes InitData and FinalData fields. InitData is any custom data type, and FinalData is any custom data type. The block synchronization event information refers to block synchronization event information related to orphan blocks.

3. The data acquisition and processing method for blockchain according to claim 1, characterized in that: The collection of new block information in step S1 is achieved through a data instrumentation method. The data instrumentation method uses the TraceData data structure to collect new block information; The TraceData data structure includes InitSnapshot and FinalSnapshot fields.

4. The data acquisition and processing method for blockchain according to claim 3, characterized in that: The data instrumentation method uses a universal interface for its instrumentation output.

5. A data acquisition and processing system for blockchain, used to implement the data acquisition and processing method for blockchain as described in any one of claims 1-4, characterized in that: This includes a data aggregation agent module and multiple data reporting service modules; The data reporting service module obtains new block information by reading and listening to the log output of blockchain nodes and sends the new block information to the data aggregation agent module; The data aggregation proxy module performs data aggregation and processing on the received new block information.

6. A data acquisition and processing system for blockchain according to claim 5, characterized in that: The data reporting service module uses a passive listening method to read and monitor the log output of the blockchain nodes.

7. A data acquisition and processing system for blockchain according to claim 5, characterized in that: The data aggregation proxy module includes a receiving submodule and multiple data processing service submodules, and the data processing service submodules correspond one-to-one with the data reporting service module. The receiving submodule distributes the new block information to the corresponding data processing service submodule according to the information source of the new block information.

8. A data acquisition and processing system for blockchain according to claim 7, characterized in that: The receiving submodule includes a Receiver Server, an MQ Server, and a Meta Server; The Receiver Server receives new block information from the data reporting service module and sends the new block information to the MQ Server and Meta Server. After receiving the new block information, the MQ Server communicates with the Meta Server to confirm the real-time performance and integrity of the data. After confirming that there are no errors, the MQ Server transmits the new block information to the corresponding data processing service sub-module according to the corresponding information between the data reporting service module and the data processing service sub-module. After receiving the new block information, the Meta Server determines whether packet loss has occurred and decides whether to notify the data reporting service module to retransmit.

9. A data acquisition and processing system for blockchain according to claim 7, characterized in that: The data processing service submodule processes the new block information according to the data processing data model; The data processing service submodule stores the processed data obtained from processing the new block information in the data storage service module. The data storage service module is equipped with a data access interface.

Images