Mass data processing system and method

Abstract

The invention discloses a mass data processing system and method. The system comprises an integration unit, a storage unit, an analytic unit and a revealing unit, wherein the integration unit is used for acquiring source data from different data sources and generating subject data of a uniform interface which corresponds to different traffics after the source data is processed; the storage unit is used for storing the subject data in distributed file systems; the analytic unit is used for carrying out multi-node and multitask parallel computation and analysis on the data stored in the distributed file systems according to different task requests; and the revealing unit is used for revealing a computing and analyzing result of the analytic unit. According to the invention, processing of mass data in an intelligent cell can be realized and the processing efficiency and the system extensibility are enhanced.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a massive data processing system and method. Background technique [0002] At present, with the construction of intelligent communities, various supporting systems (such as electricity consumption information collection system, household energy management system, distributed energy management system, electric vehicle management system, etc.) are also being constructed and improved simultaneously. In the future, in the promotion process of smart communities, smart power consumption, distributed clean energy and electric vehicles will be widely used, and a large number of information systems will be put into operation at the same time; , there will be a large number of power consumption terminal equipment and new energy access, which will inevitably generate a large amount of basic power consumption information data. [0003] Business Intelligence (BI) system provides tech...

AI Technical Summary

Problems solved by technology

At present, using a single server to store and process more than 100TB of data is not only costly, but also low in data processing efficiency

[0006] The traditional business intelligence architecture is generally implemented by the mature "ETL (Extract, Transform and Load) tool + data warehouse software + report tool", which supports a maximum of TB (1024GB) level Data storage and analysis, but cannot handle the data storage and calculation of up to PB (1024TB) level in the intelligent community, and because the existing mature data warehouse software is stand-alone, when the data in the data warehouse grows to exceed the maximum capacity of the current machine When the limit is limited, it cannot be solved by simply increasing the number of machines, but can only be replaced by machines with better performance and more expensive prices, resulting in poor hardware scalability and high equipment costs

Images