Design of background automatic computing engine of data management system

Abstract

The invention provides a design of a background automatic computing engine of a data management system, and relates to the field of MDM background computing engines. According to the design of the MDM background automatic computing engine, the technology of combining spark and hadoop distributed architecture is adopted to process large-scale data calculation. The design comprises the following specific implementation steps that a data production party produces data in kafka; an offline architecture adopts flume to collect data in the kafka to the hdfs; a real-time architecture adopts sparkstreaming to consume the data in the kafka, performs formatting processing on the original data, and writes the data into hbase; an sql script is written to format original data in the hdfs, and finally, the original data is written into an offline data bin hive by adopting snappy compression; and the formatted data is analyzed and processed, and then a final result is written into the hbase to be called by a foreground service system. Compared with a previous MDM system, the problem that data cannot be stored can be solved only by adding a server even if the data volume is large nowadays, and meanwhile, multi-path data writing can be realized, so that the user experience is better.

Description

technical field [0001] The invention relates to the field of MDM background computing engines, in particular to the design of a data management system background automatic computing engine. Background technique [0002] MDM (Mobile Device Management), also known as mobile device management, provides complete life cycle management of mobile devices from device registration, activation, use, and elimination. Mobile device management (MDM) can realize functions such as user and device management, configuration management, security management, and asset management. Mobile device management (MDM) can also provide comprehensive security system protection, while managing and protecting mobile devices, mobile apps, and mobile documents. [0003] In the existing technology, the MDM background cannot store a large amount of data, and cannot support large-scale data calculation, resulting in poor user experience, and the data writing path is single, resulting in relatively large limit...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the deficiencies of the prior art, the present invention provides the design of the background automatic calculation engine of the data management system, which solves the problem that the storage system cannot store large data

Images