Code stream data management method and device and storage medium

Abstract

The invention discloses a code stream data management method and device and a storage medium, and the method comprises the steps of analyzing to-be-stored data, and obtaining a data identification and a timestamp of the to-be-stored data; generating a data file name and an index file name of the to-be-stored data according to the data identifier; generating a partition storage path of the to-be-stored data according to the timestamp; performing partition storage on the to-be-stored data according to the data file name, the index file name and the partition storage path; and when a data query request is received, performing data retrieval in response to the data query request. Thus, partition storage is performed according to the analysis result of the code stream data, equipment resources are greatly saved, the operation and maintenance complexity is reduced, the index file is constructed during data storage, and when the code stream data needs to be queried, the required data can be quickly searched from the corresponding storage space according to the characteristics of partition storage in combination with the index file, so that the query response time is effectively accelerated, and the user perception is obviously improved.

Description

technical field [0001] The present invention relates to the technical field of information processing, in particular to a code stream data management method, device and computer-readable storage medium. Background technique [0002] In the era of big data, with the rapid development of technologies such as mobile networks, the Internet, the Internet of Things, and the Internet of Vehicles, various network devices and various user terminals are developing explosively. Massive data will be generated in real time between terminals and devices, and between devices. stream data. The characteristics of code stream data are: various types, large data volume, and fast daily growth rate. For a small subsystem, the total code stream of a day may be TB level, or even PB level. Backtracking the original code stream is an inevitable requirement. How to efficiently store and quickly reverse the code stream is becoming more and more difficult. [0003] The existing solutions for accessin...

AI Technical Summary

Problems solved by technology

However, the overhead of the cluster itself during HBase storage is high, resulting in high resource usage during storage, too long task startup time during query, and untimely task response

In order to improve the code stream access efficiency of HBase and other big data clusters, it is necessary to continuously increase storage and computing equipment, which eventually leads to the continuous increase in the product cost and operation and maintenance cost of the HBase cluster itself, while the user experience continues to decline

Images