Service dual-active implementation method and system of big data platform

Abstract

The invention discloses a service dual-active implementation method and system of a big data platform. The method comprises that a Hadoop cluster massive file processing cluster obtains and stores source files and preprocesses source data in the source files; a first cluster processes the data, thus obtaining data processing results and carries out node blocking to the data processing results; related data of the data processing results in the first cluster is synchronized to a second cluster by a synchronization tool; the first cluster and the second cluster are equipped with same numbers of nodes; and node-to-node increment synchronization is carried out between the first cluster and the second cluster by binary data. According to the MPP database architecture based on column storage provided by the invention, the node-to-node increment synchronization is carried out between the homogeneous clusters by the binary data; through transmission of compressed increment file blocks, the batch inquiry is separated; moreover, the online inquiry timeliness is ensured; and the problems of time and resource conflicts between batch processing and online real time inquiry are solved.

Description

technical field [0001] The present invention relates to the technical field of Internet and cloud computing, and more specifically, to a method and system for realizing active-active business of a big data platform. Background technique [0002] With the continuous development of the Internet and cloud computing technology, more and more services need to use cluster services to better meet user needs. On the one hand, service clustering enables users to get a better user experience, such as fast access, and at the same time, service clustering can provide services for more users at the same time; on the other hand, service clustering can improve system availability , to reduce the harm caused by system failure to the business. [0003] At present, with the continuous advancement of big data construction, on the basis of better solutions to data integration and processing, as the demand for data in the operation process increases, the service capabilities of big data platfor...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to propose a method and system for implementing active-active business on a big data platform to solve the problem in the prior art that batch processing and data query cannot be performed simultaneously due to the fact that data cannot be read and written at the same time during batch processing

Images