Space data analysis method for semi-structured big data

Abstract

The invention belongs to the field of data analysis technology, and discloses a space data analysis method for semi-structured big data. The data are collected through a data collection module. Operation statements which are input by a user and are of extraction and counting for the semi-structured big data are received through a client module, and a parsing and conversion module is synchronized with the operation statements for processing. The operation statements are received through the parsing and conversion module, and are parsed, and a parsing result is converted into a configuration rule. An application engine module is called through the client module according to the configuration rules to generate job tasks, a bottom-layer framework and a cluster are combined with a processing process as assistance to realize extraction and counting command operations on the semi-structured big data and result processing with the help of a client, and maintainability and automation and visualization levels are improved. A complete method is provided for semi-structured big-data operations, a unified model is used for service and analysis scenes, data redundancy is reduced, and operating,maintenance and operations of service data analysis are simpler, more convenient and more reliable.

Description

technical field [0001] The invention belongs to the technical field of data analysis, and in particular relates to a spatial data analysis method for semi-structured big data. Background technique [0002] At present, structured data is also called row data, which is logically expressed and realized by a two-dimensional table structure, strictly follows the data format and length specifications, and is mainly stored and managed through a relational database. The opposite of structured data is unstructured data that is not suitable for representation by two-dimensional database tables, including office documents in all formats, XML, HTML, various reports, pictures and audio, video information, etc. Databases that support unstructured data use multi-value fields, field fields, and variable-length field mechanisms to create and manage data items, which are widely used in full-text retrieval and various multimedia information processing fields. Commonly used big data analysis c...

AI Technical Summary

Problems solved by technology

[0004] To sum up, the problems existing in the existing technology are: the existing spatial data analysis method for big data is cumbersome and needs to go through processes such as configuration (different business scenarios), execution and viewing, and cannot be automatically processed after executing commands like SQL queries , and there are also defects in the visualization of the execution process and running results

Images