Unlock instant, AI-driven research and patent intelligence for your innovation.

Query method and query device for column storage files

A query method and column storage technology, which are applied to the query method of column storage files, query devices, computer equipment and readable storage media, can solve the problems of directly querying column storage formats that do not support SPL statements, and achieve convenient query , the effect of expanding the scope of the query

Active Publication Date: 2019-08-27
PING AN TECH (SHENZHEN) CO LTD
View PDF6 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Therefore, the present invention aims to solve the problem that the SPL statement does not directly query the column storage format

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Query method and query device for column storage files
  • Query method and query device for column storage files
  • Query method and query device for column storage files

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0044] Refer to figure 1 , Shows a flow chart of the steps of the method for querying stored files in the first embodiment of the present invention. It can be understood that the flowchart in this method embodiment is not used to limit the order of execution of the steps. It should be noted that, in this embodiment, the query device 2 for column storage files (hereinafter referred to as the query device 2) is used as the execution subject for exemplary description. details as follows:

[0045] Step S100: Obtain the SPL query sentence input by the user from the terminal.

[0046] Specifically, when the user needs to query the column storage file, the query device 2 obtains the SPL query sentence input by the user from the terminal. Wherein, the query sentence at least includes: query time range and name.

[0047] Step S102: Determine the query range in the first file of HDFS according to the SPL query sentence, where the first file is a column storage file, and the first file is cl...

Embodiment 2

[0062] See figure 2 , Shows a schematic diagram of the hardware architecture of the query device in the second embodiment of the present invention. The query device 2 includes, but is not limited to, a memory 21, a processing 22, and a network interface 23 that can communicate with each other through a system bus. figure 2 Only the query device 2 with the components 21-23 is shown, but it should be understood that it is not required to implement all the illustrated components, and more or fewer components may be implemented instead.

[0063] The memory 21 includes at least one type of readable storage medium, the readable storage medium includes flash memory, hard disk, multimedia card, card type memory (for example, SD or DX memory, etc.), random access memory (RAM), static memory Random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disks, optical disks,...

Embodiment 3

[0067] See image 3 , Shows a schematic diagram of program modules of the column storage file query system of the third embodiment of the present invention. In this embodiment, the column storage file query system 24 may include or be divided into one or more program modules. The one or more program modules are stored in a storage medium and executed by one or more processors to The present invention is completed, and the above query method for column storage files can be realized. The program module referred to in the embodiment of the present invention refers to a series of computer program instruction segments capable of completing specific functions, and is more suitable for describing the execution process of the column storage file query system 24 in the storage medium than the program itself. The following description will specifically introduce the functions of each program module in this embodiment:

[0068] The obtaining module 201 is used to obtain the SPL query sente...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a query method for column storage files. The query method comprises the following steps: acquiring an SPL query statement input by a user from a terminal; determining a query range in a first file of a distributed file system according to the SPL query statement; screening out a second file from the first file according to the query range; converting the SPL query statement into an SQL statement according to a preset conversion rule; importing the second file into a big data platform SQL search engine to enable a big data platform SQL search engine to execute the SQL statement so as to search a target query file, with the big data platform SQL search engine comprising Hive and / or Spark SQL; and outputting the target query file to the terminal. According to the query method for the column storage files provided by the embodiment of the invention, a unified query mode is provided for a user of an original log search system, the query range of SPLstatements is expanded, and convenience is provided for query of column storage data.

Description

Technical field [0001] The embodiments of the present invention relate to the technical field of database management, and in particular to a query method, query device, computer equipment and readable storage medium for column storage files. Background technique [0002] In the current log search system, the Search Processing Language (SPL) developed by Splunk is a common retrieval language used to query log data that has been indexed. However, sometimes due to disk space requirements, log data with a long storage time will be stored in the distributed file system (Hadoop Distributed File System, HDFS) in the form of column storage (such as parquet or optimized row columnar (orc)). ) To save space. When you need to query these data, you are required to use SPL statements to query the data files in these column storage formats. However, data files in the current column storage format often only support the use of Structured Query Language (SQL) as a query engine for query statem...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/18G06F16/182G06F16/14
CPCG06F16/1815G06F16/182G06F16/148Y02D10/00
Inventor 陈俊峰
Owner PING AN TECH (SHENZHEN) CO LTD