Data structured analysis method based on elasticsearch

Abstract

The present invention relates to a data structure analysis method based on ElasticSearch, including constructing JSON Schema according to the JSON data of the document to be written into ElasticSearch; detecting the associated ClickHouse data table in the metadata of the target index; Write the mapping relationship of the columns of the data table into the data table; synchronize the data of the data table with the data of ElasticSearch; use structured tools to query and analyze the stored data through the data table. On the premise of maintaining the text retrieval ability, the present invention writes the data into ClickHouse synchronously, provides the structural analysis ability, and the technical staff can conveniently perform statistical query, correlation query, multi-dimensional aggregation and analysis on the numerical data, reducing the system cost. The difficulty of development and maintenance improves the research and development efficiency and usability of the system.

Description

technical field [0001] The invention belongs to the technical field of industrial Internet platform data analysis, and relates to a data structure analysis method based on ElasticSearch. Background technique [0002] The industrial Internet platform is a platform that can realize the rapid interconnection of enterprise products, production equipment and systems. The platform can help customers deploy comprehensive and flexible business processing capabilities through data analysis and machine learning, and help enterprises achieve digitalization, Intelligent development. For the public industrial Internet platform, its service goal is to involve all walks of life, so the platform is required to have strong versatility; in order to meet this versatility, the platform processes and forms basically adopt a schema-free data structure method, specifically: ElasticSearch data structure, which supports distributed, multi-tenant and full-text search, has excellent read, write and r...

AI Technical Summary

Problems solved by technology

[0003] The industrial Internet platform needs to analyze the data and use the data analysis to guide the customer's deployment. However, although the ElasticSearch used by the current platform can better complete data reading and writing, access and text-based retrieval, the data in ElasticSearch is stored in JSON format. Lack of model constraints, the analysis method is mainly based on text retrieval, and the standardization and quality of data depend on the software system design and the quality of relevant technical personnel

When performing data analysis, it is necessary to query the document JSON data from ElasticSearch, and then transmit it to the application service, and perform data analysis, calculation and analysis in the application service, and data quantitative analysis capabilities (including data statistics, association query, group aggregation, etc.) Weaker, users need to independently design and implement the JSON data obtained according to the query results, which increases the complexity of system design and implementation

Moreover, ElasticSearch's data transmission is in JSON format, and the data storage overhead is relatively large. Each data record document transmitted carries complete format field data. The data transmitted by the network is relatively large, consuming more network bandwidth and memory overhead for data processing.

It is only suitable for the retrieval of text data, not conducive to the association query and aggregation analysis of numerical data

Images