Method and system for processing data in real time

Abstract

The invention discloses a method and a system for processing data in real time. The method comprises receiving data in real time and putting the data into a queue, reading the data in the queue circularly and putting the read data into a cache, determining whether the time of reading data from the queue exceeds a preset time threshold, and if so, aggregating the data in the cache and persisting the aggregated data. According to the method and the system for processing data in real time, the time threshold is preset; when the queue data are processed, jumping out of the cycle of the processing queue is carried out at a fixed time interval, and therefore, regular persistence can be guaranteed, the data of the same dimensionality can be aggregated as less as possibly, the contradictory relation of the persistence pressure, the persistence period and the aggregation effect can be balanced, and various negative effects due to large data size or different speeds of data going in and out of the queue can be avoided.

Description

technical field [0001] The invention relates to the technical field of data persistence, in particular to a real-time data processing method and system. Background technique [0002] In a real-time data processing system, on the one hand, a large amount of data with dimensions will enter the queue according to a fixed time window; on the other hand, when the queue is processing data, it is expected to quickly aggregate data with the same dimension in the same time window into The number of entries should be as few as possible. In the case of a particularly large number of dimensions, the number of aggregated entries is also huge. Considering the pressure of persistent aggregated data on the server, it is also necessary to minimize the amount of data persisted each time, that is, It is said that when the data in a time window has not been fully put into the cache, the data that has been put into the cache needs to be aggregated and persisted first. [0003] The existing real...

AI Technical Summary

Problems solved by technology

When the speed of fetching data from the queue exceeds the speed of data entering the queue, the existing technology may cause an aggregation and persistence immediately after each piece of data is fetched, and the persistence will be particularly frequent

And when the speed of fetching data in the queue is lower than the speed of data entering the queue, it may cause a large backlog of data in the queue. If the data in the queue cannot be fetched all the time, the data in the cache cannot be aggregated and persisted. The processing results are no longer real-time. If the data in the queue is finally empty after a long backlog, the amount of data after aggregation of the data in the cache will usually be more. Instantly the pressure will increase

Images