Data structured analysis method based on ElasticSearch

Abstract

The invention relates to a data structured analysis method based on ElasticSearch. The method comprises the following steps: constructing JSON Schema according to JSON data of a document to be written into ElasticSearch; detecting an associated ClickHouse data table in the metadata of the target index; writing the data information into the data table according to the mapping relation between the fields and the columns of the data table; enabling the data of the data table to be synchronized with the data of the ElasticSearch; and querying and analyzing the stored data through the data table by using a structured tool. On the premise of keeping the text retrieval capability, the data is synchronously written into ClickHouse, the structured analysis capability is provided, a technician can conveniently perform statistical query, associated query, multi-dimensional aggregation and analysis on numerical data, the development and maintenance difficulty of the system is reduced, and the research and development efficiency and usability of the system are improved.

Description

technical field [0001] The invention belongs to the technical field of industrial Internet platform data analysis, and relates to a data structured analysis method based on ElasticSearch. Background technique [0002] The industrial Internet platform is a platform that can realize the rapid interconnection and intercommunication of enterprise products, production equipment and systems. The platform can help customers deploy comprehensive and flexible business processing capabilities through data analysis and machine learning, and help enterprises realize digitalization, networking, Intelligent development. For the public industrial Internet platform, its service goal is to involve all walks of life, so the platform is required to have strong versatility; in order to meet this versatility, the platform processes and forms basically adopt a modeless data structure method, specifically ElasticSearch data structure, this type of data structure supports distributed, multi-tenant...

AI Technical Summary

Problems solved by technology

[0003] The industrial Internet platform needs to analyze the data and use the data analysis to guide the customer's deployment. However, although the ElasticSearch used by the current platform can better complete data reading and writing, access and text-based retrieval, the data in ElasticSearch is stored in JSON format. Lack of model constraints, the analysis method is mainly based on text retrieval, and the standardization and quality of data depend on the software system design and the quality of relevant technical personnel

When performing data analysis, it is necessary to query the document JSON data from ElasticSearch, and then transmit it to the application service, and perform data analysis, calculation and analysis in the application service, and data quantitative analysis capabilities (including data statistics, association query, group aggregation, etc.) Weaker, users need to independently design and implement the JSON data obtained according to the query results, which increases the complexity of system design and implementation

Moreover, ElasticSearch's data transmission is in JSON format, and the data storage overhead is relatively large. Each data record document transmitted carries complete format field data. The data transmitted by the network is relatively large, consuming more network bandwidth and memory overhead for data processing.

It is only suitable for the retrieval of text data, not conducive to the association query and aggregation analysis of numerical data

Images