On-chain transaction data fast parsing method for utxo model blockchain

Abstract

The present application belongs to the technical field of block chain, and specifically relates to a chain transaction data rapid analysis method for a UTXO model block chain. The present application mainly comprises the following steps: according to the data structure of the block chain, directly analyzing corresponding original binary data files to obtain each piece of structured chain transaction data of the currency; obtaining the input address and amount of each transaction from the chain transaction data and supplementing the input address and amount to the input field of the original transaction; and finally, supplementing the spending output transaction hash and spending time field to the output field of the original transaction. The present application comprehensively uses a key-value database and a file reading and writing method, can rapidly and synchronously analyze the data of UTXO block chains such as Bitcoin, Bitcoin Cash, Litecoin, Dogecoin and D0GE from the first transaction to the latest transaction, and supplement the key fields in the transaction input and output. The present application can be extended to the rapid analysis process of the chain transaction data of more UTXO block chains, and provides support for data analysis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of blockchains, and particularly relates to a method for rapidly analyzing on-chain transaction data of a UTXO model-oriented blockchain. BACKGROUND

[0002] Since Nakamoto proposed the Bitcoin system based on the UTXO data model in 2008, various types of blockchain system applications have emerged. The source code of the Bitcoin system is open, so anyone can modify it. Today, among the top 20 blockchains in terms of market value, many are based on the Bitcoin system source code and use the UTXO (Unspent Transaction Output) model. The model is characterized by recording the process of each transaction, and the output of each transaction will be spent as input by the subsequent transaction.

[0003] As the market size of blockchain system applications continues to expand, more and more users are participating in it. While new industries such as exchanges have emerged in the blockchain application field, some new types of crime and illegal behavior have also begun to appear, and regulatory agencies need to analyze, monitor, and provide early warning for malicious behavior and risk events. On-chain data of public blockchains is an important basis for analyzing blockchain data, but on-chain data has the characteristics of large data volume and fast data addition speed. How to efficiently and accurately obtain full on-chain data is a challenge. Currently, there are three main methods for obtaining on-chain transaction data: browser API query, local RPC interface call, and binary file parsing.

[0004] 1. Blockchain data query browsers provide query interfaces API for on-chain transaction data, but these APIs are limited by website performance and cannot be used in large-scale data analysis.

[0005] 2. After synchronizing data by locally building a full node client, users can use the RPC interface provided by the client to obtain transaction data. Although this method is more efficient than browser API calls, it requires frequent calls to the local RPC interface, especially when obtaining all transactions within a certain block. Each transaction requires a corresponding RPC interface call to obtain transaction details. This way of obtaining and analyzing data is inefficient.

[0006] 3. After setting up a full node, the on-chain transaction data obtained synchronously is stored in binary file format. Direct parsing can achieve higher performance than the two methods mentioned above. Several tools already use this method, such as BlockSci and BlockETL. However, BlockSci's limitation lies in its reliance on an in-memory database, resulting in high computational resource requirements and poor data storage reliability. BlockETL, on the other hand, uses the relational database MySQL and only supports parsing Bitcoin transaction data. Applying it to UTXO blockchains with larger single-block capacities may cause the tool to malfunction due to the maximum value limit of a single field and single record in the MySQL engine.

[0007] This invention defines a standardized data parsing and storage interface, applicable to various UTXO model blockchain data, and exhibits strong scalability. Furthermore, this invention stores the parsing results in file format, allowing users to freely choose further persistence methods according to their needs, thus offering high flexibility. Summary of the Invention

[0008] To address the aforementioned problems—namely, the time-consuming nature of obtaining full transaction data from public blockchains and the difficulty in acquiring transaction data from blockchains with various UTXO models—this invention provides a general method for rapidly parsing transaction data from UTXO model blockchains. The invention employs the following technical solution:

[0009] This invention provides a method for fast parsing of on-chain transaction data in UTXO model blockchains. The method is used to quickly parse on-chain transaction data in UTXO model blockchains, where the on-chain transaction data is stored in multiple binary files. The method includes the following steps:

[0010] Step S1: Based on the data structure information of the UTXO model blockchain, multiple binary files are read simultaneously by multiple processes and parsed into unlinked structured transaction data, which are then stored as several transaction result files for the current day, in units of date.

[0011] Step S2: Traverse the transaction result file of the day in chronological order, store the UTXO output of each transaction in the key-value database, then traverse each transaction in the transaction result file of the day, find the corresponding UTXO for the input cost of the transaction in the key-value database, fill the transaction input details field of the transaction in the transaction result file of the day according to the information of the found UTXO, and write the information of the UTXO to the spending status file in units of date;

[0012] Step S3, filling the transaction output details field of each transaction in the corresponding daily transaction result file according to the information in the spent condition file, completing the analysis and bidirectional linking of the on-chain transaction data.

[0013] The UTXO model blockchain-oriented on-chain transaction data rapid analysis method provided by the application can also have the following technical features: the serial number in the file name of the binary file corresponds to the time sequence of the transaction; in step S1, multiple binary files are read and analyzed in parallel by using multiple processes; when assigning the task of reading the binary files to each process, the following assignment rule is used: for m binary files and n processes, n < m; the binary files are sorted by file name; the first n-1 processes are assigned to read ⌊m / n⌋ binary files in turn, and the last process reads all the remaining binary files.

[0014] The UTXO model blockchain-oriented on-chain transaction data rapid analysis method provided by the application can also have the following technical features: in step S1, after each process reads a binary file, all transactions in the block contained in the binary file are organized according to date, and transactions belonging to the same day are written into the same daily transaction result file; each process independently performs the analysis of the binary file and the writing of the daily transaction result file; when writing the daily transaction result file, all writing is append writing.

[0015] The UTXO model blockchain-oriented on-chain transaction data rapid analysis method provided by the application can also have the following technical features: in step S2, all daily transaction result files are traversed; for each daily transaction result file, all transaction information in the file is read and stored in memory, and all UTXOs of the read transaction outputs are stored in the key-value database; then all transactions of the day in memory are traversed, the corresponding UTXO of each transaction input is found in the key-value database, the UTXO is deleted from the key-value database, and the input address and amount are filled into the transaction input details field of the transaction in memory.

[0016] The UTXO model blockchain-oriented on-chain transaction data rapid analysis method provided by the application can also have the following technical features: in step S2, multiple processes are used to traverse the transactions in a daily transaction result file.

[0017] The UTXO model blockchain on-chain transaction data rapid analysis method provided by the application can also have the following technical features: in step S3, the transaction hash and the spending time of the transaction output details field of each transaction in the current day transaction result file are filled according to the information in the spent condition file.

[0018] The UTXO model blockchain on-chain transaction data rapid analysis method provided by the application can also have the following technical features: in step S3, the transaction output details field of the current day transaction result file of different dates is filled by using multi-process.

[0019] Inventive action and effect

[0020] According to the UTXO model blockchain on-chain transaction data rapid analysis method provided by the application, according to the data structure of the UTXO model blockchain, the original binary file is directly parsed and stored as multiple local current day transaction result files according to date, and the characteristics of the binary file content organization are used to simultaneously read multiple different files by using multi-process, so that the parsing rate can be improved; and the bidirectional link of the transaction information in the local current day transaction result file is completed by using the chain value database in combination with the local file, which has higher performance compared with the bidirectional link of the transaction information realized by reading and modifying in the relational database in the prior art.

[0021] In summary, the application directly parses the binary file of the UTXO model blockchain, reasonably divides the transaction data storage file, and can realize rapid analysis of full-amount transaction data of multiple currencies, thereby providing a scheme for quickly obtaining data for users who want to supervise and analyze the UTXO model blockchain. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a schematic diagram of the UTXO model blockchain on-chain transaction data rapid analysis method in the embodiment of the application;

[0023] Figure 2 is a flowchart of the UTXO model blockchain on-chain transaction data rapid analysis method in the embodiment of the application;

[0024] Figure 3 is an example diagram of the UTXO model blockchain binary file storage method in the embodiment of the application;

[0025] Figure 4 is an example diagram of the current day transaction result file storage structure in the embodiment of the application;

[0026] Figure 5 is an example diagram of the transaction detailed information in the embodiment of the application;

[0027] Figure 6 is an example diagram of the file storage mode after the transaction input detail field is linked in the embodiment of the present application;

[0028] Figure 7 is an example diagram of the transaction detailed information after the transaction output detail field is linked in the embodiment of the present application;

[0029] Figure 8 is an example diagram of the UTXO spending detailed information for the transaction output detail field linking step in the embodiment of the present application;

[0030] Figure 9 is an example diagram of the transaction detailed information after the transaction output detail field is linked in the embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the present application of the on-chain transaction data rapid analysis method for the UTXO model block chain is specifically described below in combination with embodiments and drawings.

[0032] <EMBODIMENT>

[0033] Figure 1 、 Figure 2 are a schematic diagram and a flowchart of the on-chain transaction data rapid analysis method for the UTXO model block chain in the embodiment.

[0034] As shown in Figures 1-2 , the on-chain transaction data rapid analysis method for the UTXO model block chain specifically includes the following steps:

[0035] Step S1, according to the data structure information of the UTXO model block chain, the on-chain transaction data of the currency is parsed into structured transaction data, and is stored as a plurality of local daily transaction result files in units of dates.

[0036] The full node client synchronizes the on-chain data stored in the local binary files. According to the parameter settings of the client, the size of each binary file is usually not more than 128 MB, and the file name format is blkxxxxx, where xxxxx is Arabic numerals, and the insufficient digits are filled with 0. After connecting to other peer nodes, the full node client will start to synchronize the on-chain data to the local, and the binary files synchronized to the local are stored from blk00000 files. When the file size reaches 128 MB, a new blk00001 file is created for continuous writing, and so on. The binary files store the information of each block and the information of each transaction in each block. Since the full node client uses multiple processes to synchronize data from multiple peer nodes, most of the block information in the same binary file is sorted in chronological order, but there are a small number of blocks out of order. Understanding this organization rule can read these binary files, directly parse the read data into structured transaction data and persist it as a local file.

[0037] Step S1 specifically includes:

[0038] Multiple binary files of on-chain data are read simultaneously using multiple processes, and are parsed according to the predetermined data structure of the UTXO blockchain. Considering that there may be a small amount of disorder but mostly in order in the block data in the binary file, when assigning reading file tasks to each process, the following rules are used: for m binary files and n processes, n < m, the binary files are sorted by file name, and the first n-1 processes are assigned to read ⌊m / n⌋ binary files, and the last process reads all the remaining binary files. For example, assuming there are 10 binary files and 3 processes, process 1 is assigned to read binary files 1, 2 and 3, process 2 is assigned to read binary files 4, 5 and 6, and process 3 is assigned to read binary files 7, 8, 9 and 10. This allocation scheme can greatly ensure that the blocks parsed by each process are far apart, providing convenience for the subsequent step of writing local files.

[0039] After each process reads a binary file, all transactions in the blocks contained in the binary file are organized in date form, and transactions belonging to the same day are written into the same daily transaction result file. According to the above process task allocation scheme, since the height difference between the blocks read by each process is large, the probability of any two processes writing a file at the same time is extremely small. Even if two processes write to the same file, the writing method used in this scheme ensures the atomicity of each line of data writing, that is, multiple process writing transactions will only cause the order of writing transactions to change, and will not cause a single transaction to cross with other transaction data. The subsequent linking operation reads all transactions of a day into memory, so the order of transaction data in the file has no effect. And all write operations are append write, thus avoiding the defect of low efficiency of random write.

[0040] Each process independently performs binary file parsing and daily transaction result file writing, and the input and output of any process do not depend on the execution of another process, thereby obtaining good concurrent effect. After all processes are parsed, a local daily transaction result file of full-amount transactions in date units is obtained, and each yyyy-mm-dd date form folder includes a csv file storing information of all transactions of the day.

[0041] Figure 3 is an example diagram of the UTXO model blockchain binary file storage method in this embodiment.

[0042] As shown in Figure 3 In this embodiment, taking synchronous parsing of transaction data as an example, first, the original binary file of the full node synchronization transaction data is obtained, and is stored as blk00000.dat, blk00001.dat, blk00002.dat, and so on.

[0043] Figure 4 is an example diagram of the daily transaction result file storage structure in this embodiment.

[0044] As shown in Figure 4 In this embodiment, multiple processes are enabled to read the above multiple binary files, each process is assigned a different binary file for reading and parsing, and the detailed information of each transaction read is stored in the csv file under the corresponding date folder according to date units. The structured transaction result file obtained by multi-process parsing forms a file structure tree of Figure 4 In the folder named by year, there are multiple folders named by date, and in each folder named by date, there is a result.csv file.

[0045] Figure 5is an example graph of transaction details in this embodiment.

[0046] As shown in Figure 5 , the csv file uses a semicolon separator, and each transaction is composed of 9 fields. Figure 5 Taking one transaction in the csv file as an example, the meanings of the fields are shown in Table 1:

[0047] Table 1. Field meaning table of transaction file

[0048] Field Name Field Value Version Number 2 Transaction Size in Bytes 219 Transaction Lock Time 0 Number of Transaction Inputs 1 Number of Transaction Outputs 2 Transaction Input Details [{"sequence”:0}…………] Transaction Output Details [{"address”:bitcoincash:qz7g}…………] Transaction Timestamp 1657418363 Transaction Hash 6a7bd579 …………

[0049] After step S1, the structured transaction data is parsed and stored as a local daily transaction result file, as shown in Figure 5 . At this time, the daily transaction result file has not been bidirectionally linked, and the transaction input details field in the transaction data is missing the input address (“address”) and the amount (“value”), and the transaction output details field is missing the transaction hash of the spent UTXO (“spent_tx”) and the spending time (“spent_timestamp”). Only the UTXO spent by the transaction input comes from the first output of the previous transaction, which uniquely determines a UTXO. Therefore, these fields need to be filled in the following steps.

[0050] Step S2, fill in the transaction input details field of the structured transaction data obtained in step S1, traverse the daily transaction result file in chronological order, store the UTXO output by each transaction in the key-value database, and then traverse each transaction in the daily transaction result file, find the corresponding UTXO in the key-value database, fill in the transaction input details field of the transaction according to the information of the found UTXO in the daily transaction result file, and write the UTXO information to the local spent file.

[0051] Step S2 specifically includes:

[0052] A local key-value database is built as a container for storing UTXO, and the UTXO is deleted from the key-value database after being spent, thereby saving memory.

[0053] In order to avoid missing, the contents of the daily transaction result file of a day need to be read first, the read transaction information is stored in the memory, and the outputs of all transactions are stored in the key-value database. Then, all the transactions of the day in the memory are traversed, the corresponding UTXO of each transaction input is found in the key-value database, the UTXO is deleted from the key-value database, and the transaction input details field of the transaction information in the memory is modified. The transaction input details field is filled in at the same time as the information of the transaction completing a UTXO is stored as a spent file in the unit of date, which lays the foundation for filling the transaction output field in step S3.

[0054] In the process of traversing the daily transaction result file of each day, multi-process acceleration is adopted, and the transactions of the day are divided into multiple processes to traverse and fill the transaction input details field. The mechanism of the UTXO account model ensures that there is no double-spending problem, that is, the same UTXO cannot be spent twice, so when each process modifies the UTXO stored in the key-value database, there is no conflict between multiple processes operating on the same UTXO.

[0055] Figure 6 is an example diagram of the file storage method after the transaction input details field is linked in this embodiment.

[0056] As shown in Figure 6 , after step S2, the transaction input details field is filled in the result.csv file under each date folder, and the utxo.csv file recording the UTXO spending information of the day is generated in the same folder.

[0057] Figure 7 is an example diagram of the transaction detailed information after the transaction input details field is linked in this embodiment.

[0058] As shown in Figure 7 , after step S2, the “address” and “value” fields and their corresponding values have been filled in the transaction input details field.

[0059] Step S3, fill the transaction output details field of the structured transaction data obtained in step S2, fill the transaction output details field in the corresponding daily transaction result file according to the information in the spent file generated in step S2, so as to complete the analysis and bidirectional linking of on-chain transaction data.

[0060] After step S2, the transaction hash and the spending time of each output of each transaction are obtained, and the information is stored in the form of a local file. In step S3, the local transaction result file of the day is updated according to the UTXO spending file, so as to complete the analysis and bidirectional linking of the full transaction. All operations in this step can be realized by local file reading and writing, and there is no sequential dependence relationship. The files in the file folder of different dates can be operated by using multi-process to complete the filling of the transaction output detail field.

[0061] Figure 8 FIG. 6 is an example diagram of the UTXO spending details used in the transaction output detail field linking step in this embodiment.

[0062] As shown in FIG. 6, the UTXO spending details are stored in the utxo.csv file, and are separated by semicolons. Each row has three fields. The first field represents the spent UTXO, which is uniquely represented by the transaction hash plus # plus the nth output n. The second field is a number representing the timestamp of the spent UTXO. The third field represents the transaction hash of the UTXO spending. The information is filled into the result.csv file in the same folder to complete step S3. Figure 8

[0063] FIG. 7 is an example diagram of the transaction details after the transaction output detail field linking in this embodiment. Figure 9 As shown in FIG. 7, after step S3, the “spent_tx” and “spent_timestamp” fields and their corresponding values are filled in the transaction output detail field in the result.csv file, indicating the spending situation of the output of the transaction.

[0064] Figure 9 At this point, the linking work of the full transaction data is completed, and is stored in the form of a local file according to the date. The data can be stored in a database for subsequent query and analysis.

[0065] In this embodiment, the above method is used to analyze and link the full on-chain transaction information from January 3, 2009 to July 14, 2022. The time consumption is shown in Table 2.

[0066] Table 2 Time consumption table of analyzing and linking full transaction data

[0067] Table 2 Time consumption table of analyzing and linking full transaction data

[0068] Phase Time Spent (hours) Parse binary to get full unlinked transaction file 5 Input field linking 13 Output field linking 3 Total 21

[0069] It can be seen that the analysis and linking of the full on-chain transaction information takes only 21 hours, which proves that the method of the present application is fast and efficient.

[0070] ​Effects of embodiments

[0071] According to the method for quickly analyzing on-chain transaction data of the UTXO model blockchain provided by the embodiment, according to the data structure of the UTXO model blockchain, the original binary file is directly analyzed and stored as a plurality of local transaction result files of the same day according to date, and the characteristics of the binary file content organization are utilized, and a plurality of different files are read simultaneously by using multi-processes, so that the analysis rate can be improved. Then, the chain value database is combined with the local files to complete the bidirectional linking of the transaction information in the local transaction result file of the same day, compared with the prior art of reading and modifying in the relational database to realize the bidirectional linking of the transaction information, the performance is higher.

[0072] Specifically, in step S1, according to the characteristics of the binary file content organization, the memory usage, the number of transactions in one day and other values are comprehensively considered, and the above process allocation rule is adopted, so that the probability that two processes read different binary files and then write the transaction result file of the same day in the same date folder is greatly reduced. Even if this situation occurs, the use of atomic writing method can ensure the integrity of the transaction data written by multiple processes when writing into the same file, so that the analysis process has high concurrent performance, and all the writing file operations are append writing, which has high efficiency advantage and can avoid the defect of low efficiency of random file writing.

[0073] Further, in step S2, since the mechanism of the UTXO account model ensures that there is no double-spending problem, in step S3, the filling operation of the output field is local file reading and writing and has no sequential dependency, so multi-processes can also be used in steps S2 and S3 to improve the bidirectional linking efficiency.

[0074] The above embodiments are only used to illustrate the specific implementation of the present application, and the present application is not limited to the description range of the above embodiments.

Claims

1. A method for fast parsing on-chain transaction data of a UTXO model blockchain, used for fast parsing on-chain transaction data of a UTXO model blockchain, the on-chain transaction data is stored as a plurality of binary files, characterized in that, The method comprises the following steps: Step S1, according to the data structure information of the UTXO model blockchain, simultaneously reading multiple binary files and parsing into unlinked structured transaction data by using multiple processes, and storing as a plurality of daily transaction result files in units of dates, wherein the serial number in the file name of the binary file corresponds to the time sequence of the transaction, and when assigning each process to read the binary file, the following assignment rule is adopted: for m binary files and n processes, n < m, the binary files are sorted by file name, and the first n-1 processes are assigned to read m / n binary files in turn, and the last process reads all the remaining binary files; Step S2, traversing all the daily transaction result files in time sequence, reading all the transaction information in each daily transaction result file and storing in the memory, and storing the UTXO output of each transaction in the key-value database, then traversing each transaction in the daily transaction result file, searching for the corresponding UTXO spent by the transaction input in the key-value database, deleting the UTXO from the key-value database and modifying the transaction input detail field of the transaction in the memory, filling in the input address and amount, filling in the transaction input detail field of the transaction in the daily transaction result file according to the information of the found UTXO, and writing the information of the UTXO into the spent file in units of dates; Step S3, filling in the transaction output detail field of each transaction in the corresponding daily transaction result file according to the information in the spent file, completing the parsing and bidirectional linking of the on-chain transaction data.

2. The UTXO model blockchain-oriented on-chain transaction data rapid parsing method according to claim 1, characterized in that: wherein In step S1, after each process reads a binary file, all transactions in the block contained in the binary file are organized in date form, and the transactions belonging to the same day are written into the same daily transaction result file, Each process independently performs the parsing of the binary file and the writing of the daily transaction result file, When writing the daily transaction result file, all writing is append writing.

3. The UTXO model blockchain-oriented on-chain transaction data rapid parsing method according to claim 1, characterized in that: wherein In step S2, multiple processes are used to traverse the transactions in a daily transaction result file.

4. The UTXO model blockchain-oriented on-chain transaction data rapid parsing method according to claim 1, characterized in that: wherein, In step S3, the transaction hash and spending time of the transaction output detail field of each transaction in the corresponding daily transaction result file are filled according to the information in the spent file.

5. The UTXO model blockchain-oriented on-chain transaction data rapid parsing method according to claim 1, characterized in that: wherein In step S3, the filling of the transaction output details field is performed for the said day's transaction result files of different dates using multi-processing.

Images