Time slice processing method based on trumpet

Abstract

The invention discloses a time slice processing method based on a trumpet, and the method can enable the length of a time slice to be similar to the trumpet according to the definition of a user, enables the length of the time slice closer to the current time to be shorter, and enables the length of the time slice longer than the current time to be longer. The trumpet-based time slice processing method is mainly divided into four parts: definition of trumpet-shaped time slices, creation of the trumpet-shaped time slices, management of the trumpet-shaped time slices and query of the trumpet-shaped time slices. According to the trumpet-shaped time slicing method, data with a larger time span can be reserved, a large amount of storage space is saved, index query delay under the large time span is further reduced, and query results can still be returned within millisecond-level delay.

Description

technical field [0001] The invention relates to the field of network information processing, in particular to a processing method based on trumpet-shaped time slices, and provides a data processing and storage method for dynamically adjusting the length of time slices as time goes by. Background technique [0002] In the fields of risk control, marketing, etc., related flow aggregation calculation problems such as "a person's total transaction volume in the past 24 hours" and "a person's maximum transaction amount in the past week" are often involved. Since time will always move forward and more transaction flow will enter the system, aggregation calculations under such constraints as "the past 24 hours" and "the past 1 week" need to consider the fact that time has been moving, so that It is necessary to slice the time, and then aggregate and calculate the flow of each slice. [0003] At present, the processing techniques for time slices are mainly divided into three catego...

AI Technical Summary

Problems solved by technology

Data can be stored in memory to ensure that the query delay is within a few to tens of milliseconds, but the memory capacity is limited, which will cause the service to be unable to accommodate so much data, resulting in abnormal service operation;

[0010] 2) A large amount of data can also be stored on the disk, but the disk access speed is slow, resulting in high latency for index calculation and query.

[0011] In addition, frequent disk reads and writes can also cause serious performance jitter problems;

[0012] 3) When querying, the system needs to re-aggregate these slices. Due to too many time slices involved, the query delay is very high, and it is difficult to meet the high-frequency access to this part of the data scenario (such as ultra-low latency, ultra-high concurrent real-time risk control scenarios) performance requirements

[0015] 1) The system can save the original flow to ensure the accuracy of user queries, but the storage space it occupies is uncontrollable. When faced with high-frequency scenarios, the system will occupy too much storage space and cause excessive calculation and query delays , unable to meet the requirement of millisecond delay;

[0016] 2) The system only calculates the index of the session time slice and discards the original flow, then the user will not be able to get the query result and cannot meet the business requirements

[0017] In summary, the above three traditional time slicing techniques cannot simultaneously meet the following requirements:

Images