Electric power big data quality real time monitoring method

Abstract

The invention discloses an electric power big data quality real time monitoring method. The electric power big data quality real time monitoring method comprises steps of inputting enterprise application data flow in real time, dividing the data flow into multiple batches, adopting DStreams to perform stream-oriented computation, using Spark batch processing to execute conversion, Spark frameworktask scheduling and memory management and outputting a batch processing result. The electric power big data quality real time monitoring method uses the electric power big data quality real time monitoring method based on the flow-oriented computation technology to realize a whole process quality management on power supply enterprise data according to an electric power enterprise data lifecycle, improves data quality, guarantees accuracy, timeliness, effectiveness and reliability of the data and provides a strong guarantee to data integration and mining application.

Description

technical field [0001] The invention relates to the field of data quality monitoring, in particular to a method for real-time monitoring of power big data quality. Background technique [0002] The key to big data processing is to solve data quality problems. Only by avoiding data errors and ensuring data quality can enterprises benefit from big data applications. Ensuring data quality is a prerequisite for big data to bring value to enterprises. Big data analysis is inseparable from data quality and data management. High-quality data and effective data management can ensure the authenticity and value of analysis results, whether in academic research or in commercial applications. [0003] In recent years, the informatization of the power industry has also made great progress, from the initial automation of electric power production to the construction of management informatization represented by financial computerization in the 1980s, and then to the large-scale enterprise ...

AI Technical Summary

Problems solved by technology

However, the data quality and the level of data sharing and utilization are not high. First, the data support for analysis and decision-making is low. The same data has multiple sources and inconsistent statistical standards. Second, the data support for operation management needs to be improved, and the data quality varies. Part of the data has no business system support, and lacks unified norms, standards, and clear data accountability; third, data quality control lags behind, and the control work is one-sided, and a complete data quality control system and comprehensive and effective data quality have not been formed. Guarantee mechanism to restrict the deep mining of data value

Images