Big data processing method and system based on multiple types of devices and readable medium

Abstract

The invention discloses a big data processing method and system based on multi-type devices and a readable medium, a big data platform composed of an ETL layer, a service layer and a storage layer is constructed, the ELT layer comprises a data acquisition module and a data support module, the service layer comprises a Stream module, a batch processing module and a GPU resource module, and the storage layer comprises a distributed storage module. The method comprises the following steps: enabling original data to flow into a big data cluster or a storage layer through a data cleaning pipeline, carrying out consumption and real-time calculation on real-time data in the original data through a Stream module, and carrying out data cleaning on offline data in the original data through a batch processing module; and finally, storing the data in a distributed storage module, and classifying and publishing the data in combination with a data support module. According to the method, the complex data treatment process returned by the sensing equipment can be programmable and expansible, and the problems that the pumped storage power station capital construction site data is difficult to manage and calculate and the utilization rate is low are thoroughly solved.

Description

technical field [0001] The present invention relates to the field of big data, in particular to a big data processing method, system and readable medium based on multi-type equipment. Background technique [0002] In recent years, with the continuous popularization of IoT devices, more and more data has been produced and collected. Especially in the construction site of the pumped storage power station, the data generated by the sensing equipment of the Internet of Things began to expand continuously, the types of data increased, and the complexity of the data became higher, which not only made the maintenance of the construction site face unprecedented challenges, but also ushered in Huge opportunity. Therefore, how to collect and use these data with low latency and high efficiency has become an issue of widespread concern at present. A multifunctional big data platform that can perform standardized data collection, complex protocol analysis, massive data calculation, dis...

AI Technical Summary

Problems solved by technology

[0005] In view of the above-mentioned problems such as difficult management, difficult calculation and low utilization rate of the on-site data of the infrastructure construction of the pumped storage power station

Images