Self-adaptive rate control method for stream data processing

Abstract

The invention belongs to the technical field of computer applications and relates to a self-adaptive rate control method for stream data processing. According to the method, based on a common data receiving message queue and a big data distributor calculating framework, the degree of parallelism of data processing is adjusted through a pre-fragmentation mode according to the condition of a current calculating colony and the quantity of current processed data of the colony is dynamically adjusted according to a self adaptive real time rate control method, so that the stability of the calculating colony is ensured, and the delay of data stream processing is reduced. Along with gradual penetration of big data into the industries, the application range of the real-time processing of mass data is gradually expanded. The real time property and the stability of a mass data processing system are quite important. On the premise of not increasing the quantity of the colony hardware and the task programming complexity, the stability and the processing efficiency of the calculating colony are enhanced to a certain extent.

Description

technical field [0001] The invention belongs to the technical field of computer applications, and relates to an adaptive rate control method for streaming data processing. Background technique [0002] At present, with the development of technology, the amount of data is increasing day by day, and "big data" technology has penetrated into all walks of life. At present, many devices will collect a large amount of data, and it is hoped to process the data in time to discover its value. For example, data generated by smartphones, sensors, Internet of Things devices, social networks, and online transaction systems need to be continuously collected and analyzed in real time to achieve rapid response. Therefore, how to improve the ability of real-time data analysis and processing has become a very important issue. [0003] Some current mainstream big data real-time processing frameworks include Spark, Storm, Flink, etc. Spark Streaming is an extension of the Spark core API, whi...

AI Technical Summary

Problems solved by technology

However, discarding data usually leads to inaccurate calculation results. Dynamic resource management requires more hardware resources for data processing peaks. Dynamic batch size makes running delays uncertain.

[0005] The state of the cluster is not static, so the artificially set upper limit of the static data processing rate may be inaccurate. If it is too small, it may not be able to fully utilize computing resources when the rate of processing bursts increases. If it is too large, the system may receive too many data, causing high latency in the computing cluster, resulting in system instability

Images