Data modeling and storage method of line network big-data of rail transit command center

Abstract

The invention discloses a data modeling and storage method of line network big-data of a rail transit command center. According to the adopted data modeling and storage organization method, structureddata in the line network big-data are combined with characteristics of a Hadoop platform and components and data application situations of a rail transit industry to select a reasonable data model ina modeling process of the structured data, data modeling and data storage are carried out, and detailed description is specially carried out for a method of logic modeling: temporary data files are stored in Hbase, historical data after unified modeling are stored in the Hbase, and tables are established for an intermediate data layer and a data mart layer by a method of paradigm reducing and arestored in Hive; and for unstructured data, a Hadoop platform is adopted to store small files according to application and time classifications, and full-text retrieval and analysis of the unstructured data are realized. Therefore, standardized and highly efficient data storage is realized for the line network big-data.

Description

technical field [0001] The invention relates to a data modeling and storage method for rail transit command center line network big data, and belongs to the technical field of rail transit monitoring systems. Background technique [0002] With the acceleration of urban rail transit construction, the subway lines in various cities are gradually developing towards the network. There are more and more data types and data volumes in the rail transit network. Massive data are collected in the rail transit command center. . In the field of rail transit, how to effectively collect, organize, store, and even process and analyze these structured and unstructured data, conduct in-depth data mining and data analysis, and mine valuable information, so as to improve the operation level of rail transit , improve scientific decision-making capabilities, improve efficiency and reduce costs, and improve information services and security capabilities have increasingly become the focus of the...

AI Technical Summary

Problems solved by technology

However, there is no index in Hadoop's data storage, and the data blocks are much larger than MPP DB, so the speed of precise data query and table-to-table combination query is slower than that of MPP DB. In the past, MPP DB's data modeling method based on paradigm modeling It cannot be copied to the Hadoop platform, so it is necessary to select the appropriate components according to the characteristics of the Hadoop platform, and design a new combination of dimensional modeling and paradigm modeling for the structured data in the rail transit line network big data Data modeling methods to avoid the disadvantages of the Hadoop platform and maximize its advantages to achieve safe and efficient storage and access of network big data

[0005] In the past, the unstructured data of the rail transit command center, such as video, image, voice, log files, page crawling, etc., were all stored in disk arrays, which only realized the storage backup function, and the full text of unstructured data could not be realized. retrieval for further analysis

Images