HIVE-based natural gas pipeline data processing method and data warehouse system

Abstract

The invention relates to an HIVE-based natural gas pipeline data processing method and a data warehouse system. The HIVE-based natural gas pipeline data processing method comprises five steps of determination of a natural gas pipeline system data source, processing and fusion of multi-source data, integration of a data warehouse and mining of demand data. According to the method, the multi-source heterogeneous data of the target data dimension is obtained by determining the data source and the metadata of the natural gas pipeline system, and the multi-source heterogeneous standard data of the target data dimension is obtained by converting and synthesizing the related fields under the data source to be processed. And finally, establishing a data warehouse, and extracting required data including natural gas pipeline operation pressure and flow, compressor inlet and outlet pressure, pressure ratio and the like as required based on the established data warehouse. According to the method, useful data information can be easily extracted from mass data, the extraction process of traditional data is simplified, efficient utilization of pipeline system operation data is achieved, and technical support is provided for analyzing the operation state of a natural gas pipeline system.

Description

technical field [0001] The invention belongs to the field of multi-source data processing of natural gas pipelines, in particular to a HIVE-based natural gas pipeline data processing method and a data warehouse system. Background technique [0002] Oil and gas resources are important energy minerals and strategic resources, as well as important chemical raw materials. In addition, natural gas has the advantages of being clean, environmentally friendly, economical and safe, and has a wide range of uses. Pipeline transportation is currently the most common, economical, and safest transportation method for long-distance natural gas on land. [0003] The natural gas pipeline transportation system is a continuous and closed system. Usually, the SCADA system is used to collect and record the flow parameters during the natural gas pipeline transportation process. The natural gas pipeline flow parameter data information recorded by the SCADA system is an intuitive view of the natura...

AI Technical Summary

Problems solved by technology

Therefore, it is necessary to adopt a method to establish a data warehouse system for natural gas pipelines, so as to solve the problems of data dispersion and low overall efficiency, and meet the needs of natural gas pipeline information construction and efficient management. Provide services and provide technical support for improving the comprehensive management capabilities and safe operation capabilities of natural gas pipelines

[0004] To sum up, the current natural gas pipeline data has the characteristics of discreteness, massiveness, and authenticity, but there are problems such as low comprehensive management efficiency, and there is a lack of mature and efficient data warehouses for comprehensive management and efficient storage of pipeline historical operation data. It is impossible to conveniently maintain and coordinate the utilization of pipeline data information

Images