Data archiving method, system and equipment and storage medium

Abstract

The invention provides a data archiving method, system and equipment and a storage medium. The data archiving method comprises: acquiring performance parameters of service database equipment; Judgingwhether the performance parameters meet a preset filing condition or not; If yes, judging that the system meets the requirement;, if yes, obtaining the service type of each to-be-archived service table from the database; and obtaining filing configuration according to the service type, the filing configuration setting a corresponding relationship between each service table and a filing table in acurrent filing task, generating a filing instruction according to the filing configuration, and executing the filing instruction to store each service table of a database into the filing table in theHBase distributed storage system. According to the method, the characteristic of the HBase dynamic column is utilized, multiple business table models can be supported to be combined into one distributed table, new fields can be expanded at will, the coupling relation between the database business table and the filing table can be reduced, redundant data is reduced, and therefore the storage spaceis reduced, and data access logic is simplified.

Description

technical field [0001] The present invention relates to the Internet field, in particular to a data archiving method, system, equipment and storage medium. Background technique [0002] In a large-scale application system, when the data volume of some business tables in the database develops rapidly, there will be an explosive growth of millions of new data items every day. As a file system, the database system will have longer response time and poor stability when the amount of internal data increases, seriously affecting the read and write efficiency of business tables and the operation of the system. Separating reads and writes from the database, sub-databases and sub-tables is only a temporary solution, not a permanent solution, because as the amount of data gradually increases, the access performance of the database will still be affected. [0003] Adopting a certain archiving strategy, regularly extracting historical data from business tables for archiving, and cleari...

AI Technical Summary

Problems solved by technology

[0004] The above prior art is mainly to create a special historical data table in the database, migrate the historical data in the database to the historical data table, and delete the historical data from the original database, thereby reducing the pressure on the data when the database is running, but the business The model changes quickly, and after the model changes, it needs to be synchronized to the historical archive table in time

At the same time, in the above method, a large amount of historical data is still stored in the database, and the query of historical data is still done by reading the database, which will still put a lot of pressure on the database, and there is a problem of long database response time

Images