Highly available and scalable distributed data archiving method and system

Abstract

The invention provides a highly available and scalable distributed data archiving method and system, which can realize archiving algorithm configuration and schedule task templating, and the flexible configuration of data collection algorithms; and can also realize the dynamic expansion of scheduling servers and the stable performance. The method includes the steps of configuring and caching archiving task information, wherein, the caching of the archiving task information includes centralized cache of the archiving task information and local cache in schedule servers; registering an archiving task into the Spring to generate templated schedule tasks; loading the schedule tasks by a scheduling center, and allocating the schedule tasks to the plurality of schedule servers; and obtaining, by the schedule servers, the corresponding archiving task information according to the allocated schedule tasks for data archiving.

Description

technical field [0001] The invention relates to the fields of computer technology and software, in particular to a highly available and scalable distributed data archiving method and system. Background technique [0002] With the development of science and technology, modern society has entered the era of big data, especially in the Internet industry. With the development of business and the increase of users, the amount of data on the business system platform is increasing, which eventually leads to increased pressure on the database and poor system responsiveness. , User experience decline. [0003] Through the user operation analysis and data analysis of the system, it can be found that there is a large amount of inactive data or non-hot data. The occupancy rate of these data in some systems even reaches 80%-90%, which seriously affects the performance and stability of the system. Therefore, effective data archiving methods are needed to archive and process data in a tim...

AI Technical Summary

Problems solved by technology

[0006] The data collection algorithm is not flexible, if the collection requirements change, the code needs to be modified;

[0007] Scheduling tasks lack templates and are too rigid. Customized development is required for different scheduling tasks. Different table structures correspond to a scheduling task. The development cost is high and the code volume is heavy, which is not conducive to maintenance;

[0008] The scalability of the scheduling server is poor, and it needs to be modified at the code level to realize task distribution and redeployment

Images