Performance benchmark test system and method for big data stream processing framework

Abstract

The invention relates to a performance benchmark test system and method for a big data stream processing framework. The system comprises a streaming load generator, a streaming scenario and application constructor, a performance data collection tool and a performance data analysis tool. The invention selects an application that conforms to the streaming processing mode calculation feature, generates a load that conforms to the streaming processing mode data feature, tests the performance of the big data stream processing framework under typical scenarios and applications, and collects performance indicators such as back pressure, throughput, delay, system resources, and node data during operation. Finally, the bottlenecks of the stream processing framework are diagnosed by analysis and statistics of collected data.

Description

technical field [0001] The invention relates to a performance benchmarking system and method of a large data stream processing framework, in particular to the performance performance of the framework in typical streaming scenarios and applications, and belongs to the field of software technology. Background technique [0002] With the advent of the Internet era and the continuous development of technologies such as mobile Internet, social networks, and e-commerce, data has shown explosive growth. Big data has become a hot spot in today's scientific and technological circles, business circles, and even the government. [0003] In general, data can be divided into bounded data and infinite data. Bounded data, also called batch data, refers to fixed and bounded data stored in persistent media, and the amount of data does not change during calculation. Generally speaking, the batch big data processing framework (hereinafter referred to as the batch processing framework) receive...

AI Technical Summary

Problems solved by technology

However, these works all have problems such as the application test set coverage is too small

[0008] To sum up, the existing performance benchmarks for stream processing frameworks have three deficiencies in terms of application test sets, streaming data sources and performance indicators. One is that they do not support dynamically changing data sources, and the other is that the structure of the application is too simple. , the feature coverage of streaming processing is low. Third, most of the performance indicators only consider delay and throughput, and do not involve other indicators such as back pressure.

Images