Offline analysis method for massive data

A technology for off-line analysis and massive data, applied in the field of off-line analysis, can solve problems such as increasing data robustness and cleanliness, achieve the effect of solving the bottleneck of data collection, improving collection efficiency, and improving efficiency

Inactive Publication Date: 2016-07-20
STATE GRID CORP OF CHINA +3
View PDF3 Cites 40 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Most of these data are stored in the Oracle system that is closely integrated with the business. How to effectively analyze and utilize these

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Offline analysis method for massive data

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0041] Example

[0042] The invention utilizes the distributed storage of massive data of the Hadoop platform cluster and the efficient and fast parallel computing capability, specifically:

[0043] Data collection and preprocessing. For different power big data types, diversified data collection modes can be used. For streaming data, the Kafka collection tool is used to aggregate the streaming data into the Kafka cluster, and then store and process it by HBase. For the relational database, the Sqoop data exchange tool is used, combined with the customizable data cleaning module, and the distributed import of relational data to HDFS or HBase is realized through the Map-Reduce distributed computing framework. At the same time, Sqoop provides an incremental import function of data. For big data files, you need to import them into Hadoop using FTP protocol or localized upload. After the electric power big data is stored in HDFS, the data can be cleaned, converted, filtered and oth...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an offline analysis method for massive data. The method comprises the following steps: (1) collecting massive power data; (2) preprocessing the massive power data; (3) offline analyzing the massive power data; and (4) displaying an analysis result. By adopting the offline analysis method for the massive data provided by the invention, centralized storage, unified management and sharing of power data resources of different times and spaces, different business and different scenes can be realized; and scientific and reasonable reference is provided for power decision of a management layer by the analysis and the mining of the massive historical data.

Description

technical field [0001] The invention relates to a method for off-line analysis, in particular to a method for realizing off-line analysis of massive data. Background technique [0002] Electric power information technology is developing in the direction of intelligent integration of data and information applications, and its operation mode will move towards a new stage centered on services. Smarter. Big data is the concentrated expression of technology-integrated development and intelligent application concepts under the new situation. Value-added content Value-added service as the goal of the application model, its core is the development and utilization of information resources. [0003] With the continuous deepening and advancement of smart grid construction, the amount of data generated by power grid operation and equipment inspection / monitoring is increasing exponentially, especially the massive heterogeneous and polymorphic data generated in the process of distributi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 潘森周爱华朱力鹏饶伟黄进蔡皓
Owner STATE GRID CORP OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap