Data loading and storing equipment, method and system

A data loading and storage device technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of huge memory, insufficient data compression support, etc., to save storage space, data reading speed and data query The effect of increasing speed

Inactive Publication Date: 2016-08-24
LETV INFORMATION TECH BEIJING
View PDF6 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the prior art, the solution to data storage is to use traditional relational databases or non-relational databases to store data, but they have the following disadvantages when facing massive data storage: 1. Due to the A better query speed can only be achieved when the ratio of data volume to memory is 3:1. Therefore, in the face of massive data, if you want to get better query speed, you need a huge memory; 2. Due to insufficient data compression support, therefore, Need more storage space to complete data storage

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data loading and storing equipment, method and system
  • Data loading and storing equipment, method and system
  • Data loading and storing equipment, method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] Specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to illustrate and explain the present invention, and are not intended to limit the present invention.

[0029] figure 1 Schematic diagram of the structure of the data loading, storage and query system provided by the present invention. like figure 1 Described, the present invention provides a kind of data loading, storage and query system, and this system comprises: ApacheKafka cluster 100, is used to receive data from more than one data source; Data loading and storage device 200 (for example, Apache Hadoop cluster); And Apache The Solr cluster 300 is configured to receive query conditions, obtain the path and row ID of the RCfile format file of the corresponding data according to the query condition, and obtain corresponding data according to the path and row ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the field of data receiving, storage and query, discloses data loading and storing equipment and method and a data loading, storage and query system, and can obviously improve the storage performance and the query performance of data. The data loading and storage equipment comprises a storage unit, a reading unit and a data loading unit, wherein the storage unit is used for storing an RCfile format file; the reading unit is used for reading the data; and the data loading unit is used for reading index field data from the data according to a metadata definition, generating an index file according to the index field data, the path of the RCfile format file and the ID (Identity) of a row used for storing the data in the RCfile format file, and storing the data and the index file, as row records, into the RCfile format file.

Description

technical field [0001] The present invention relates to the field of data storage and query, in particular to a data loading and storage device, method, and data loading, storage and query system. Background technique [0002] In the Internet and communication industry, users are generating massive amounts of behavioral data (such as Internet records, call records, etc.) all the time, and how to effectively store and query massive data is a very important problem currently facing. [0003] In the prior art, the solution to data storage is to use traditional relational databases or non-relational databases to store data, but they have the following disadvantages when facing massive data storage: 1. Due to the A better query speed can only be achieved when the ratio of volume to memory is 3:1. Therefore, in the face of massive data, if you want to get a better query speed, you need a huge memory; 2. Due to insufficient support for data compression, therefore, More storage spa...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2291
Inventor 李锋标
Owner LETV INFORMATION TECH BEIJING
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap