Management method and device for processing resources in distributed stream processing

Abstract

The invention provides a management method for processing resources in distributed stream processing. The method comprises the steps that when a processing load of a certain processing unit meets a predetermined splitting condition, a maximum object processing rate of the processing unit is acquired; statistics is performed on an object generation rate of each object stream in an object stream group processed by the processing unit; the object stream group comprises at least one object stream; at least one subset of the object stream group is made into a new object stream group, and the new object stream group is processed by other processing units, wherein the sum of the object generation rates of all the object streams in each subset does not exceed an object processing rate peak value; and the object processing rate peak value is determined according to the maximum object processing rate. Through the technical scheme, the object stream group is automatically split according to the object processing rates and the object generation rates when a stream task is busy to increase the stream processing rate, and processing units responsible for processing can be added in time according to running conditions when the volume of business is increased.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a method and device for managing processing resources in distributed stream processing. Background technique [0002] In a distributed stream processing system, stream processing is distributed among multiple processing units in the form of tasks. Each processing unit corresponds to an object stream group, and the object stream group may include one or more object streams, and the objects in these object streams will be the processing objects of the corresponding processing unit. Taking the Storm system as an example, when a topology starts running, the components of the topology (spout or bolt) will be parallelized into multiple tasks and assigned to multiple Workers (that is, processing units) for execution; the task input is Partition (message Stream, a kind of object stream) in a steady stream of messages (a kind of object), each Worker can be assigned ...

AI Technical Summary

Problems solved by technology

In this way, for some business flows with irregular peak or trough periods, if it is necessary to use limited processing resources to achieve guaranteed speed object processing, system managers often need to manually adjust the processing based on real-time monitoring of business volume The number of processing units corresponding to the business flow not only has a huge workload, but also for sudden changes in the business volume, it is often difficult to adjust in time due to the long time-consuming manual operation

Images