Flow aggregation method and device and electronic equipment

Abstract

The invention provides a flow aggregation method and device and electronic equipment, and the method comprises the steps: responding to a received first flow aggregation instruction, obtaining a firstmessage of a first message queue and a second message of a second message queue indicated by the first flow aggregation instruction, and respectively carrying out the flow aggregation of the first message and the second message according to a time sequence; checking whether the processing time difference of the first message and the second message reaches a preset time difference threshold valueor not, wherein the processing time difference is the difference between the event time carried by the first message and the event time carried by the second message in processing; if yes, newly adding a thread for flow aggregation processing to the message queue with the low processing speed, or reducing the thread for flow aggregation processing of the message queue with the high processing speed; and performing association operation on the processing result of the first message and the processing result of the second message. According to the invention, the association operation of the flowaggregation processing results of the two message queues is effectively realized, and the processing results are better presented.

Description

technical field [0001] The present application relates to the field of big data analysis, in particular to a flow aggregation method, device and electronic equipment. Background technique [0002] As a message processing center, the Kafka cluster is responsible for tasks such as message storage, confirmation, and retry. The Kafka cluster contains multiple message queues (Message Queue), where messages in the message queue are generated by message producers and sent to the message processing center. . The message consumer obtains the message from the message processing center, and then processes it accordingly. Generally speaking, a device as a message consumer can perform stream aggregation (Stream Aggregate) processing on the data stream of the message queue, that is, after grouping the messages in one or more message queues based on time intervals, calculate each For messages within a time interval, common aggregation functions include SUM, COUNT, AVG, MIN, MAX, etc. ...

AI Technical Summary

Problems solved by technology

[0005] However, the tolerance delay time is often the experience value of the operator and cannot meet the actual needs

If the tolerable delay time is too long, the data volume of delayed output messages in the data stream will exceed the cache space of the device; if the tolerable delay time is too short, there may be a difference in the processing time of two data streams greater than the tolerable delay time, which is As a result, stream alignment cannot be achieved, and stream aggregation processing of two data streams at the same time cannot be achieved.

Images