Kinetic energy data processing system and method based on Spark computing framework

Abstract

The invention relates to a kinetic energy data processing system based on a Spark computing framework. The kinetic energy data processing system comprises a data acquisition module used for monitoringkinetic energy data of an industrial workshop, a data processing and analyzing module used for carrying out data computing on data streams issued to a Kafka message queue, and a distributed data storage module. Considering the variety of the data acquisition terminals, the Kafka cluster is provided with functional interfaces capable of being accessed to various data acquisition terminals, so thata flexible and extensible data acquisition system is established, and the problem of access of a plurality of data terminals is avoided. And secondly, as the amount of detection data needing to be processed is large, the stream data from the Kafka message queue is processed by adopting stream calculation, and the calculation performance can be greatly improved. And thirdly, the real-time performance of processing and analyzing the data according to the decision requirement of the industrial workshop on equipment operation is high, so that a distributed storage mode is adopted for the data.

Description

Technical field [0001] The invention relates to the technical field of kinetic energy data processing for various equipment in a workshop, and in particular to a kinetic energy data processing system and method based on a Spark computing framework. Background technique [0002] The equipment in the power workshop of a cigarette factory will generate massive amounts of kinetic energy data. In order to reduce production costs and improve production quality, it is necessary to collect, process and analyze these kinetic energy data. In the process of rapid growth of big data technology, there will be the following problems: [0003] (1) Due to the huge amount of data and the need to connect multi-directional system data, these data cannot be collected well; [0004] (2) A large amount of data will also introduce noise data that does not meet the quality requirements due to diversity, complexity, and various uncertain factors such as channel quality during the transmission process, such ...

AI Technical Summary

Problems solved by technology

[0003] (1) Due to the huge amount of data and the need to connect multi-directional system data, it is not possible to collect these data well;

[0004] (2) A large amount of data will also introduce noise data that does not meet the quality requirements due to diversity, complexity, and various uncertain factors such as channel quality during transmission, such as anomalies, errors, repetitions, and missing data. The accuracy of the results of the data analysis link is reduced;

[0005] (3) The real-time performance of terminal visualization data retrieval is not high

Images