Batch streaming computing system performance guarantee method based on queue modeling

Abstract

The invention discloses a batch streaming computing system performance guarantee method based on queue modeling. The performance guarantee method comprises five steps of dividing a process; selecting components; carrying out performance modeling; computing delay; and positioning and optimizing a bottleneck. According to the method, for the feature that load intensities have clear fluctuation in an operation process of a batch streaming computing system, key components in the batch streaming computing system are extracted, a performance model of the system is established according to a queuing theory and mathematical analysis is carried out on the model; data processing delay of the system under different load intensities is computed through utilization of the model in the operation process of the system; and when the data processing delay cannot satisfy a data processing timeliness demand, a performance bottleneck component is positioned according to the queuing theory, and a configuration optimization suggestion is provided.

Description

technical field [0001] The invention belongs to the field of distributed computing, and in particular relates to a performance analysis and optimization method of a batch streaming computing system. Background technique [0002] Streaming data is an important data type of big data, which has the characteristics of continuity, volatility, and dynamics. Big data streaming computing is an analysis and processing technology for streaming data. It takes data processing timeliness as the performance goal and quickly mines the value of streaming data. Batch streaming computing is an important branch of big data streaming computing. The core technical feature of batch streaming computing is to divide the received streaming data into multiple small batches in chronological order, and use MapReduce-like batch computing to process them periodically. Batch streaming computing has broad demand and application prospects in real-time data stream processing of IoT sensors and social netwo...

AI Technical Summary

Problems solved by technology

This is difficult to adapt to the characteristics of rapid changes in the load of the streaming system, resulting in a lag in performance guarantees, and cannot evaluate system performance in a timely manner when the load intensity changes

[0005] (2) The performance dependencies of the components in each stage are not comprehensively considered, and the performance bottleneck cannot be accurately located

When the system performance fails to meet the expected goal, the existing technology does not consider the complex performance dependencies among the components, but simply selects a key component to implement an optimization scheme, which cannot accurately locate the performance bottleneck, and cannot optimize the adopted optimization scheme. The performance evaluation after the optimization of the scheme reduces the efficiency of performance assurance

[0006] In general, there is currently no performance assurance method for batch streaming computing systems based on accurate performance evaluation and bottleneck location

Images