Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Storage and near real-time query method for time-sensitive data based on open source big data

A time-sensitive, query-based technology, applied in the field of database technology and information processing, can solve the problems that open source databases do not support secondary indexes, cannot provide second-level feedback speed, and the speed of data insertion and update is slow, so as to achieve storage space and time The effect of excellent, good horizontal expansion ability, and good error recovery performance

Active Publication Date: 2019-06-14
EAST CHINA NORMAL UNIV
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The data operations supported by HBase are very limited, and because it only supports efficient queries based on primary keys, the definition of primary keys is very important; the performance support for range queries is good, but the performance of large-scale scanning is extremely poor
In addition, frequent insertion or update of HBase will greatly affect system performance, and because it does not support secondary indexes, it currently only supports indexes on primary keys, so the query performance for non-primary keys is poor
[0022] In order to overcome the shortcomings in the existing technology that big data queries cannot provide second-level feedback speed, open source databases do not support secondary indexes, high database storage expansion rate, slow data insertion and update speed, and poor performance of analysis services based on data scanning, etc., The present invention proposes a storage and near-real-time query method based on open source big data to solve time-sensitive data. By fully combining time-sensitive characteristics, a data filtering strategy is designed to reduce the amount of data scanning

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Storage and near real-time query method for time-sensitive data based on open source big data
  • Storage and near real-time query method for time-sensitive data based on open source big data
  • Storage and near real-time query method for time-sensitive data based on open source big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0045] The present invention will be further described in detail in conjunction with the following specific embodiments and accompanying drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the content specifically mentioned below, are common knowledge and common knowledge in this field, and the present invention has no special limitation content.

[0046] The present invention supports the near real-time query processing of massive time-sensitive data based on open-source big data storage and near-real-time query methods for time-sensitive data. The present invention formulates an effective data storage strategy on an open-source distributed platform, utilizes efficient data indexing technology to support time-sensitive query processing, and designs a time-sensitive data storage strategy to provide guarantee for fast file location of query and realize An index based on inverted technology provides efficient file filt...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method for storage and near-real time query of time-sensitive data based on open source big data. The method comprises the steps of establishing a near-real time query processing platform having an internal storage space and an external storage space; defining a file storage strategy and performing data processing and calculating on source data files in the internal storage space so that the source data files are stored in the external storage space after being arranged according to the time-sensitive characteristics thereof; performing reverse index with the time-sensitive characteristics of the data files as the filter conditions, establishing point index and range index to generate index information and storing the index information into the external storage space and caching the information into the internal storage space; inquiring the index information and searching the point index or range index to obtain relevant file path lists, and reading source data files corresponding to query requests according to the file path lists. Fully based on the time-sensitive characteristics, the data filter strategy is designed to reduce data scanning quantity, and thus the near-real time query feedback of big data is realized.

Description

technical field [0001] The invention belongs to the field of database technology and information processing, and in particular relates to a storage and near real-time query method for time-sensitive data based on open source big data. Background technique [0002] With the development of wireless technology and the advancement of terminal equipment, the trend of data massification has been demonstrated in various industries. In the field of scientific research, such as astronomical observation data, meteorological data, ocean monitoring data, etc., with the maturity of the sensor network, the collection of these data becomes easy, resulting in the explosive growth of log information; in addition, in the field of decision-making, such as stock trading Daily transaction data, corporate reports, and Weibo data in the market are also booming. In addition to the more significant characteristics of these data in terms of quantity, the potential correlation between data also has r...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/13G06F16/14
Inventor 晁平复翁海星张弛高祎璠张蓉
Owner EAST CHINA NORMAL UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products