Mass invoicing data real-time storage and large-scale parallel access method and system

Abstract

The invention discloses a mass invoicing data real-time storage and large-scale parallel access method and system. The method comprises the steps of obtaining mass invoicing data in a MySQL database; synchronizing the massive invoicing data to a cateDB database in real time according to the Canal; and achieving high parallel access such as real-time storage of the massive invoicing data, related invoice query, detail export and the like through the crateDB database. According to the method disclosed by the invention, the data storage capacity is improved by using the crateDB database cluster, large-scale parallel access and quick query are supported, and the compatibility with invoicing system codes is high; the consistency and timeliness of the data can be ensured by using the Canal, and a code non-intruding mode can also be achieved; in cooperation with a Zookeeper cluster, server ends and client ends of a plurality of Canals are started, high-availability deployment is supported, and the service stability is improved; according to the invention, the requirements of mass data storage, high parallel access and real-time data query of the cloud invoicing system can be met.

Description

technical field [0001] The present invention relates to the field of data storage and access, and more specifically, to a method and system for real-time storage and large-scale parallel access of massive billing data. Background technique [0002] With the rapid development of Internet technology, more and more businesses are processed by cloud. At present, the number of users and invoices in the cloud billing system is very large. The billing system needs to save a large amount of customer information and product information, and realize real-time storage of massive invoice data, and support large-scale parallel access and fast query. [0003] Most of the current systems use traditional databases such as MySQL. Although they have the characteristics of excellent performance, stable service, and support for multiple development languages, with the rapid growth of data volume, from GB to TB to PB, various operations on data It is also more and more difficult, unable to meet...

AI Technical Summary

Problems solved by technology

[0003] Most of the current systems use traditional databases such as MySQL. Although they have the characteristics of excellent performance, stable service, and support for multiple development languages, with the rapid growth of data volume, from GB to TB to PB, various operations on data It is also more and more difficult, unable to meet the needs of fast query and data insertion

The noSQL database that appeared later gave up the strong transaction guarantee and relational model of traditional SQL, and focused on the high availability and scalability of the database. It does not guarantee strong consistency. It is no problem for ordinary applications, but there are still many like finance. Enterprise-level applications have strong consistency requirements, and noSQL does not support SQL statements, code compatibility is a big problem, different noSQL databases have their own API operation data, which is more complicated

Images