A data import, query and processing method based on bitmap structure

A technology of data import and bitmap structure, which is applied in the field of big data, can solve problems such as storage performance that cannot meet the demand, massive data that cannot be loaded in full, and precise weight ranking that cannot be satisfied, and achieve fast and accurate query, precise weight ranking calculation, Optimize the effect of compressed storage

Active Publication Date: 2020-12-04
BEIJING TENGYUN TIANXIA SCI & TECH CO LTD
View PDF12 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At present, there are usually two methods for counting data sets: one is to import the data into a Set data structure, and use the feature of the Set data structure that does not allow repeated elements to perform weight ranking. However, this method A large amount of storage space is required, but generally all massive data cannot be loaded into the memory. Even if external storage is used, a large number of I / O (Input / Output input / output) operations are required when writing and querying, resulting in poor performance. One is to use HyperLogLog, DataSketches and other estimation algorithms for weight ranking statistics, which have greatly improved storage and performance, but cannot meet the needs of accurate weight ranking
[0003] Furthermore, when it is necessary to perform cross-statistics on multiple massive data sets, the traditional method is to use a relational database to perform Join (connection) operations on data stored in multiple tables, but when the amount of data is particularly large, this This approach cannot meet the demand in terms of storage or performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data import, query and processing method based on bitmap structure
  • A data import, query and processing method based on bitmap structure
  • A data import, query and processing method based on bitmap structure

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0035] figure 1 A schematic diagram of a data processing system 100 based on a bitmap structure according to an embodiment of the present invention is shown. a, figure 1 The data processing system 100 based on the bitmap structure in is only exemplary. In a specific practical situation, there may be different numbers of data import servers, coordination servers, storage servers, and caches in the data processing system 100 based o...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data import, query and handling method, computing device and data handling system based on a bitmap structure, and the data import method is suitable for executing in a dataimport server. The data import method comprises the steps that a data import instruction given from a coordinator server is responded to import one piece or a plurality of pieces of original data; referring to each piece of original data, the piece of original data is conversed to obtain corresponding bitmap structure data according to a preset data handling rule; compression treatment is carriedout on each piece of acquired bitmap structure data to generate a corresponding data block, and each data block is submitted to a storage server for storage; a storage status message fed back from thestorage server is received; if the storage status message indicates that data block storage succeeds, a message of successful storage is sent to the coordinator server, so that the coordinator servercan instruct a cache server to load the data blocks from the storage server, and the failure handling is carried out on each piece of bitmap structure data.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a data import, query and processing method based on a bitmap structure, a computing device and a data processing system based on a bitmap structure. Background technique [0002] In the field of big data technology, it is a very challenging task to perform duplicate statistics on massive data sets and to perform multidimensional cross-statistics on data from multiple massive data sets. At present, there are usually two methods for counting data sets: one is to import the data into a Set data structure, and use the feature of the Set data structure that does not allow duplicate elements to do the ranking. A large amount of storage space is required, but generally all massive data cannot be loaded into the memory. Even if external storage is used, a large number of I / O (Input / Output input / output) operations are required when writing and querying, resulting in poor performance. On...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25G06F16/22
CPCG06F16/2237G06F16/2272G06F16/258
Inventor 徐岷峰
Owner BEIJING TENGYUN TIANXIA SCI & TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products