Data aggregation method based on time sequence intermediate state data structure

Abstract

The invention discloses a data aggregation method based on a time sequence intermediate state data structure. The time sequence intermediate state data structure comprises a primary key, a feature keyand data, wherein the primary key is used for associating a specific service object in a service system; load balancing is carried out on storage and calculation of data, the feature keys are used for describing feature names of specific service objects, and the data is a numerical value formed after events in a service system are processed by the feature calculation system and a calculation method of the numerical value and is used for calculating an intermediate result during time sequence intermediate state data merging operation. According to the data aggregation method, based on a time sequence intermediate state data structure, events are converted into a plurality of parts of intermediate state data, the intermediate state data are stored in corresponding cache queues, then the intermediate state data are merged according to primary keys, feature keys and timestamps of the intermediate state data, and finally the intermediate state data are stored in a storage database. According to the method, the IO load of the system can be reduced, and the calculation load of the system is reduced and reasonably distributed.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a data aggregation method based on a time series intermediate state data structure. Background technique [0002] Time-series data, also known as time-series data, is a sequence of data recorded in chronological order. Time series feature calculation is the characteristic data obtained after processing time series by calculation methods such as summation, average, variance, etc. Time series characteristics of the number of transactions in the last hour. In industries such as the mobile Internet, the Internet of Things, and financial services, timing features are widely used in business scenarios such as preventing transaction fraud, personalized recommendations, and decision-making in the event. In addition to being used as the basis for business decision-making, timing features can also be used as input to interface with rule engines and machine learning models to handle complex...

AI Technical Summary

Problems solved by technology

On the other hand, due to the increase in event concurrency, the frequency and quantity of state data synchronization increase, and the network IO load inside the system is caused

In the end, the real-time demand for feature computing can only be met by increasing the stream computing framework nodes and memory database nodes in large numbers

As a result, the cost of ownership and operation and maintenance of the feature computing system has also risen sharply

Images