Method and system for quickly analyzing streaming block chain data

Abstract

The invention discloses a method and system for rapidly analyzing streaming block chain data, and relates to the field of data processing based on block chains, and the method comprises the following operation steps: S1, obtaining streaming data response from system middleware; s2, filtering and compressing the original data; s3, data are transmitted, data check points are stored on the basis of time watermarks, decoupling classification is carried out on the data on the basis of data events, and the data are used as data states; and S4, processing and distributing the preset event according to the data state. According to the method and the system for rapidly analyzing the streaming block chain data, a specific data compression algorithm is used according to block chain streaming data characteristics, the data processing and transmission size is reduced, time watermarking and event distribution technologies are used, a traditional data processing mode is changed, the data transmission process is optimized, and the data transmission efficiency is improved. And the efficiency of data analysis is also improved.

Description

Technical Field

[0001] The present invention relates to the field of blockchain-based data processing, and in particular to a method and system for rapid analysis of streaming blockchain data. Background Art

[0002] Blockchain is a distributed ledger technology that can be used to record transactions and data and ensure that they are transparent, secure, and cannot be tampered with. The real-time data generated by the blockchain are mainly of the following types: Transaction data: On the blockchain, transactions are recorded by creating new transaction blocks. Each transaction contains information about the transaction participants, transaction amount, timestamp, etc. These transaction data are recorded in the transaction blocks on the blockchain, and due to the characteristics of the blockchain, once written, they are difficult to modify or delete; Contract execution data: Smart contracts are automated code snippets executed on the blockchain to perform predetermined operations. When the contract is triggered and executed, the relevant execution data will also be recorded on the blockchain. This data may include the input parameters of the contract, execution results, and status changes; Asset ownership data: Blockchain can be used to record asset ownership information. When an asset transfers ownership, the corresponding ownership transfer transaction will be recorded on the blockchain, thereby reflecting the change in asset ownership in real time; Authentication and authorization data: Blockchain can be used To establish a decentralized authentication and authorization system, users can use encryption keys or other authentication methods to prove their identity, so as to access specific resources or perform specific operations. These authentication and authorization data will also be recorded on the blockchain; Supply chain data: Blockchain can be used to track the entire supply chain process from production to delivery of items. Information at each step can be recorded on the blockchain, thereby achieving real-time monitoring and transparency of the supply chain; The generation and recording of these real-time data is one of the core characteristics of blockchain technology. With the application and development of blockchain, various expansion technologies have also emerged, which further increase the amount of data transmission on the chain. In order to perform real-time analysis on the chain data, a common method is to first convert the blockchain data into streaming data. The so-called streaming data is a continuous and uninterrupted data stream, and then different methods are used to process the streaming data. The traditional method relies on the idea of ​​batch data processing, and mainly stores, searches and calculates data based on sliding time window technology.

[0003] Traditional batch processing methods can meet most streaming data processing scenarios, but they still face the following limitations: Delayed processing: Batch processing needs to wait for a certain amount of data to be accumulated and then process it all at once, which will cause a certain delay, especially for applications that require real-time response; Not adaptable to changes: Batch processing is a static processing method that is difficult to adapt to dynamic changes in data. When data changes, the entire batch processing process needs to be re-executed, making it difficult for the system to quickly adapt to changes; Resource consumption: Batch processing usually requires a lot of computing resources and storage resources because it needs to process a large amount of data at one time, which may cause resource bottlenecks and performance problems; Data consistency: During batch processing, data Consistency may be affected. If an error occurs during batch execution, a rollback operation may be required, resulting in inconsistent data; Difficult to predict: Since batch processing usually processes a large amount of data, the execution time may be relatively long, and it is difficult to accurately predict the execution time, which affects the workflow and plan; Limited analysis: Batch processing technology is difficult to provide real-time analysis results, because the data can only be processed after the batch processing is completed. When real-time analysis and decision-making are required, batch processing cannot meet the needs; Excessive complexity: Processing large-scale batch data may require writing complex scripts and programs to process the data, which increases the development and maintenance costs of the system and also faces a certain risk of data loss.

[0004] Therefore, it is necessary to propose a method and system for rapid analysis of streaming blockchain data to solve the above problems. Summary of the invention

[0005] The main purpose of the present invention is to provide a method and system for rapid analysis of streaming blockchain data, which can effectively solve the problems in the background technology.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for rapid analysis of streaming blockchain data, comprising the following steps:

[0008] S1: Get streaming data response from the system middleware;

[0009] S2: Filter and compress the original data: Based on the data time, use the time series compression algorithm based on dynamic programming to compress the data to reduce data redundancy, and add a time watermark Watermark(t) to the corresponding data as a data identifier;

[0010] S3: Transmits data and saves data checkpoints based on time watermarks, decouples and classifies data based on data events as data status;

[0011] S4: Process and distribute preset events according to data status.

[0012] Preferably, in step S1, the streaming data response is Kafka and Pulsar.

[0013] Preferably, in step S2, the main process of time series compression is:

[0014] 1. Segmentation: Divide the original time series into multiple continuous segments, and the specific segments are marked by blocks;

[0015] 2. Define the cost function: used to measure the cost of merging two adjacent paragraphs, using Euclidean distance;

[0016] 3. Dynamic programming: Using the dynamic programming algorithm, we consider each paragraph step by step from the beginning to the end of the sequence and select the adjacent paragraphs with the lowest merging cost.

[0017] 4: Path reconstruction: According to the results of dynamic programming, backtrack from the end point, determine the best path to choose, and generate a compressed sequence.

[0018] A fast analysis system for streaming blockchain data, including a blockchain node and the above-mentioned method for fast analysis of streaming blockchain data, the blockchain node is electrically connected to a middleware, the middleware is electrically connected to a stream processing engine, the stream processing engine is electrically connected to a storage system and an external interface, the blockchain node is used to transmit data to the middleware, the middleware is used to obtain streaming data responses including but not limited to Kafka and Pulsar, and the stream processing engine includes data compression transformation, data decoupling and event distribution.

[0019] Preferably, the data transmitted by the blockchain node is collected based on a data collection program, the data collected by the data collection program is input into a data bus, the data bus is used to process the data, the processed data is input into an event bus, and the event bus distributes event routing to a storage subsystem, a notification subsystem, and an analysis subsystem.

[0020] Preferably, the data compression transformation includes a segmentation module, the output end of the segmentation module is electrically connected to the input end of the division module, and the output end of the division module is electrically connected to the input end of the block identification module;

[0021] The segmentation module is used to divide the original time series into multiple continuous sections, and identify specific sections with blocks.

[0022] Preferably, the output end of the segmentation module is electrically connected to a defined cost function calculation module, and the output end of the defined cost function calculation module is electrically connected to a merging module;

[0023] The defined cost function calculation module measures the cost of merging two adjacent paragraphs based on the Euclidean distance.

[0024] Preferably, the output end of the defined cost function calculation module is electrically connected to the input end of the dynamic programming module, and the output end of the dynamic programming module is electrically connected to the input end of the selection and merging module;

[0025] The dynamic programming module is based on a dynamic programming algorithm, which considers each paragraph step by step from the beginning to the end of the sequence and selects adjacent paragraphs with the minimum merging cost.

[0026] Preferably, the output end of the dynamic planning module is electrically connected to the input end of the dynamic planning introduction module, the output end of the dynamic planning introduction module is electrically connected to the input end of the backtracking module, the output end of the backtracking module is electrically connected to the input end of the nearest path planning module, and the output end of the nearest path planning module is electrically connected to the input end of the compressed sequence generation module;

[0027] The dynamic programming introduction module is used for path reconstruction. According to the result of dynamic programming, the module backtracks from the end point to determine the best path to be selected and generate a compressed sequence.

[0028] Preferably, the output end of the data compression transformation is electrically connected to the input end of the data transmission module, the output end of the data transmission module is electrically connected to the input end of the time watermark marking module, the output end of the time watermark marking module is electrically connected to the input end of the decoupling classification and storage system, the output end of the decoupling classification is electrically connected to the input end of the data state extraction unit, the data state extraction unit includes a preset time extraction module, and the output end of the preset time extraction module is electrically connected to the input end of the processing module and the distribution module.

[0029] Beneficial Effects

[0030] Compared with the prior art, the present invention provides a method and system for rapid analysis of streaming blockchain data, which has the following beneficial effects:

[0031] 1. The method and system for rapid analysis of streaming blockchain data use a specific data compression algorithm based on the characteristics of blockchain streaming data to reduce the size of data processing and transmission, and use time watermark and event distribution technology to change the traditional data processing mode, optimize the data transmission process, and improve the efficiency of data analysis.

[0032] 2. The method and system for rapid analysis of streaming blockchain data provide a mature blockchain streaming data processing solution, which comprehensively applies multiple technologies to blockchain data analysis and processing, thereby meeting the requirements of real-time blockchain analysis. It can also serve as the infrastructure of other subsystems to provide convenience for developers.

[0033] 3. The method and system for rapid analysis of streaming blockchain data can respond in time when analyzing and processing data, and will not affect the processing efficiency due to changes. It can process data in batches. When applied to applications that require real-time response, it is fast and can adapt to dynamic changes in data. It has high adaptability and will not cause resource bottlenecks and performance temperature when processing a large amount of computing resources and storage resources at one time. When batch processing is performed, the data consistency is good, so that it will not be affected, resulting in overall data consistency. When processing a large amount of data, the overall execution time is short, and the execution time can be accurately predicted, which will not affect the workflow and plan. At the same time, it can effectively provide implementation analysis results and meet the needs of users. At the same time, the overall system does not need to write complex scripts and programs to process data, so that the development and maintenance costs of the system are controllable, and it will not easily face the risk of data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the structure of the present invention;

[0035] Figure 2 It is a schematic diagram of the structure of the present invention;

[0036] Figure 3 It is a system diagram of the data compression transformation of the present invention. DETAILED DESCRIPTION

[0037] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0038] like Figure 1 As shown, a method for rapid analysis of streaming blockchain data includes the following steps:

[0039] S1: Get streaming data response from the system middleware. The streaming data response is Kafka and Pulsar.

[0040] S2: Filter and compress the original data: Based on the data time, use the time series compression algorithm based on dynamic programming to compress the data, reduce data redundancy, and add a time watermark Watermark(t) to the corresponding data as a data identifier. The main process of time series compression is:

[0041] 1. Segmentation: Divide the original time series into multiple continuous segments, and the specific segments are marked by blocks;

[0042] 2. Define the cost function: used to measure the cost of merging two adjacent paragraphs, using Euclidean distance;

[0043] 3. Dynamic programming: Using the dynamic programming algorithm, we consider each paragraph step by step from the beginning to the end of the sequence and select the adjacent paragraphs with the lowest merging cost.

[0044] 4. Path reconstruction: According to the results of dynamic programming, backtrack from the end point to determine the best path to be selected and generate a compressed sequence;

[0045] S3: Transmits data and saves data checkpoints based on time watermarks, decouples and classifies data based on data events as data status;

[0046] S4: Process and distribute preset events according to data status.

[0047] like Figure 1-3As shown, a fast analysis system for streaming blockchain data includes a blockchain node, the blockchain node is electrically connected to a middleware, the middleware is electrically connected to a stream processing engine, the stream processing engine is electrically connected to a storage system and an external interface, the blockchain node is used to transmit data to the middleware, the middleware is used to obtain streaming data responses including but not limited to Kafka and Pulsar, the stream processing engine includes data compression transformation, data decoupling and event distribution, the data transmitted by the blockchain node is collected based on a data collection program, the data collected by the data collection program is input to a data bus, the data bus is used to process the data, and the processed data is input to an event The event bus distributes the event routing to the storage subsystem, the notification subsystem and the analysis subsystem. The data compression transformation includes a segmentation module. The output end of the segmentation module is electrically connected to the input end of the division module. The output end of the division module is electrically connected to the input end of the block identification module. The segmentation module is used to divide the original time series into multiple continuous paragraphs, and identify the specific paragraphs with blocks. The output end of the segmentation module is electrically connected to a defined cost function calculation module. The output end of the defined cost function calculation module is electrically connected to a merging module. The defined cost function calculation module measures the cost of merging two adjacent paragraphs based on the Euclidean distance. The defined cost The output end of the function calculation module is electrically connected to the input end of the dynamic programming module, the output end of the dynamic programming module is electrically connected to the input end of the selection and merging module, the dynamic programming module is based on the dynamic programming algorithm, gradually considers each paragraph from the beginning to the end of the sequence, and selects the adjacent paragraph with the smallest merging cost, the output end of the dynamic programming module is electrically connected to the input end of the dynamic programming introduction module, the output end of the dynamic programming introduction module is electrically connected to the input end of the backtracking module, the output end of the backtracking module is electrically connected to the input end of the nearest path planning module, the output end of the nearest path planning module is electrically connected to the input end of the compression sequence generation module, the dynamic The planning introduction module is used for path reconstruction. According to the result of dynamic planning, it backtracks from the end point to determine the best path to be selected and generate a compressed sequence. The output end of the data compression transformation is electrically connected to the input end of the data transmission module, the output end of the data transmission module is electrically connected to the input end of the time watermark marking module, the output end of the time watermark marking module is electrically connected to the input end of the decoupling classification and storage system, the output end of the decoupling classification is electrically connected to the input end of the data state extraction unit, the data state extraction unit includes a preset time extraction module, and the output end of the preset time extraction module is electrically connected to the input end of the processing module and the distribution module.

[0048] According to the characteristics of blockchain stream data, a specific data compression algorithm is used to reduce the size of data processing and transmission, and time watermark and event distribution technology are used to change the traditional data processing mode, optimize the data transmission process, and improve the efficiency of data analysis. A mature blockchain stream data processing solution is provided, which comprehensively applies multiple technologies to blockchain data analysis and processing, so as to meet the requirements of real-time analysis of blockchain. It can also be used as the infrastructure of other subsystems to provide convenience for developers. When analyzing and processing data, it can respond in time and will not affect the processing efficiency due to changes. It can process data in batches. When applied to applications that require real-time response, the speed is fast. At the same time, it can adapt to the dynamic changes of data, has high adaptability, and will not cause resource bottlenecks and performance temperature when processing a large amount of computing resources and storage resources at one time. When batch processing is performed, the data consistency is good, so that it will not be affected, resulting in overall consistency of data. When processing a large amount of data, the overall execution time is short, and the execution time can be accurately predicted, which will not affect the workflow and plan. At the same time, it can effectively provide the analysis results of implementation, which can meet the needs of users. At the same time, the overall system does not need to write complex scripts and programs to process data, making the development and maintenance costs of the system controllable, and will not easily face the risk of data loss.

[0049] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for rapid analysis of streaming blockchain data, characterized in that: The steps include: S1: Get streaming data response from the system middleware; S2: Filter and compress the original data: Based on the data time, use the time series compression algorithm based on dynamic programming to compress the data to reduce data redundancy, and add a time watermark Watermark(t) to the corresponding data as a data identifier; S3: Transmits data and saves data checkpoints based on time watermarks, decouples and classifies data based on data events as data status; S4: Process and distribute preset events according to data status.

2. A method for rapid analysis of streaming blockchain data according to claim 1, characterized in that: In step S1, the streaming data response is Kafka and Pulsar.

3. A method for rapid analysis of streaming blockchain data according to claim 1, characterized in that: In step S2, the main process of time series compression is:

1. Segmentation: Divide the original time series into multiple continuous segments, and the specific segments are marked by blocks; 2. Define the cost function: used to measure the cost of merging two adjacent paragraphs, using Euclidean distance; 3. Dynamic programming: Using the dynamic programming algorithm, we consider each paragraph step by step from the beginning to the end of the sequence and select the adjacent paragraphs with the lowest merging cost. 4: Path reconstruction: According to the results of dynamic programming, backtrack from the end point, determine the best path to choose, and generate a compressed sequence.

4. According to claim 1, a fast analysis system for streaming blockchain data comprises a blockchain node and the method for fast analysis of streaming blockchain data, characterized in that: The blockchain node is electrically connected to a middleware, the middleware is electrically connected to a stream processing engine, the stream processing engine is electrically connected to a storage system and an external interface, the blockchain node is used to transmit data to the middleware, the middleware is used to obtain streaming data responses including but not limited to Kafka and Pulsar, and the stream processing engine includes data compression transformation, data decoupling and event distribution.

5. A fast analysis system for streaming blockchain data according to claim 4, characterized in that: The data transmitted by the blockchain node is collected based on a data collection program. The data collected by the data collection program is input into a data bus. The data bus is used to process the data. The processed data is input into an event bus. The event bus distributes event routing to a storage subsystem, a notification subsystem, and an analysis subsystem.

6. A streaming blockchain data rapid analysis system according to claim 4, characterized in that: The data compression transformation includes a segmentation module, the output end of the segmentation module is electrically connected to the input end of the division module, and the output end of the division module is electrically connected to the input end of the block identification module; The segmentation module is used to divide the original time series into multiple continuous sections, and identify specific sections with blocks.

7. A streaming blockchain data rapid analysis system according to claim 6, characterized in that: The output end of the segmentation module is electrically connected to a defined cost function calculation module, and the output end of the defined cost function calculation module is electrically connected to a merging module; The defined cost function calculation module measures the cost of merging two adjacent paragraphs based on the Euclidean distance.

8. A fast analysis system for streaming blockchain data according to claim 7, characterized in that: The output end of the cost function definition calculation module is electrically connected to the input end of the dynamic programming module, and the output end of the dynamic programming module is electrically connected to the input end of the selection and merging module; The dynamic programming module is based on a dynamic programming algorithm, which considers each paragraph step by step from the beginning to the end of the sequence and selects adjacent paragraphs with the minimum merging cost.

9. A streaming blockchain data rapid analysis system according to claim 8, characterized in that: The output end of the dynamic planning module is electrically connected to the input end of the dynamic planning introduction module, the output end of the dynamic planning introduction module is electrically connected to the input end of the backtracking module, the output end of the backtracking module is electrically connected to the input end of the nearest path planning module, and the output end of the nearest path planning module is electrically connected to the input end of the compressed sequence generation module; The dynamic programming introduction module is used for path reconstruction. According to the result of dynamic programming, the module backtracks from the end point to determine the best path to be selected and generate a compressed sequence.

10. A streaming blockchain data rapid analysis system according to claim 4, characterized in that: The output end of the data compression transformation is electrically connected to the input end of the data transmission module, the output end of the data transmission module is electrically connected to the input end of the time watermark marking module, the output end of the time watermark marking module is electrically connected to the input end of the decoupling classification and storage system, the output end of the decoupling classification is electrically connected to the input end of the data state extraction unit, the data state extraction unit includes a preset time extraction module, and the output end of the preset time extraction module is electrically connected to the input end of the processing module and the distribution module.

Images