ES cluster data multiplexing method and system, computer device and storage medium

A data reuse and computer program technology, applied in the field of data processing, can solve problems such as waste of storage space and achieve the effect of saving data storage space

Pending Publication Date: 2019-06-18
CHINA PING AN LIFE INSURANCE CO LTD
View PDF3 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

When the ELK log system needs to be combined with the Hadoop platform for data processing and analysis, because the ES cluster does not support HDFS, the data cannot be used on the ES cluster and the Hadoop platform at the same time, resulting in the data needing to be stored on the ELK system and the Hadoop platform at the same time , data is stored repeatedly, resulting in a large waste of storage space

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • ES cluster data multiplexing method and system, computer device and storage medium
  • ES cluster data multiplexing method and system, computer device and storage medium
  • ES cluster data multiplexing method and system, computer device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0049] figure 1 It is a flow chart of the steps of a preferred embodiment of the ES cluster data multiplexing method of the present invention. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0050] refer to figure 1 As shown, the ES cluster data multiplexing method specifically includes the following steps.

[0051] Step S11, receiving a data selection instruction, and selecting target data to be multiplexed in a distributed file system (HDFS) according to the data selection instruction.

[0052]In one embodiment, the target data to be multiplexed preferably refers to data that is expected to be used in both the ELK log system and the Hadoop platform, and when a data selection instruction is received, the data selection instruction can be executed in HDFS , so as to realize the selection of the target data. The ELK log system includes ES cluster, Logstash framework and Kibana framework. The Logs...

Embodiment 2

[0073] figure 2 It is a functional block diagram of a preferred embodiment of the ES cluster data multiplexing system of the present invention.

[0074] refer to figure 2 As shown, the ES cluster data multiplexing system 10 may include a selection module 101 , an acquisition module 102 , a first creation module 103 , a second creation module 104 , a query module 105 and a synchronization module 106 .

[0075] The selection module 101 is configured to receive a data selection instruction, and select target data to be multiplexed in the distributed file system according to the data selection instruction.

[0076] In one embodiment, the target data to be multiplexed preferably refers to data that is expected to be used in both the ELK log system and the Hadoop platform. When the selection module 101 receives the data selection instruction, it can be stored in HDFS Executing the data selection instruction, thereby realizing the selection of the target data. The ELK log system...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an ES cluster data multiplexing method and system, a computer device and a computer readable storage medium. The ES cluster data multiplexing method comprises the following steps: receiving and selecting to-be-multiplexed target data according to a data selection instruction; Obtaining a storage address of the target data in the distributed file system; Creating a Hive external table, and enabling the Hive external table to point to a storage address of the target data in the distributed file system; Creating a view chart corresponding to the data of the Hive and the EScluster; And receiving a data interaction request of an ES cluster, and synchronizing the interaction data from the distributed file system to the ES cluster by using the Hive external table and the view table. The invention relates to a cloud storage technology, data sharing between an ES cluster and an HDFS is realized by establishing a data transmission channel between the ES cluster and the HDFS, and repeated data storage is avoided.

Description

technical field [0001] The invention relates to the field of data processing, in particular to an ES cluster data multiplexing method, system, computer device and computer-readable storage medium. Background technique [0002] Most of the current enterprise log systems are developed based on ELK (short for Elasticsearch, Logstash and Kibana). When the ELK log system needs to be combined with the Hadoop platform for data processing and analysis, because the ES cluster does not support HDFS, the data cannot be used on the ES cluster and the Hadoop platform at the same time, resulting in the data needing to be stored on the ELK system and the Hadoop platform at the same time , Data is stored repeatedly, resulting in a large waste of storage space. Contents of the invention [0003] In view of the above, the present invention provides an ES cluster data multiplexing method, system, computer device and computer-readable storage medium, which can realize data multiplexing in th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/27G06F16/22G06F16/13G06F16/182
Inventor 石晓龙黄望饶鑫
Owner CHINA PING AN LIFE INSURANCE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap