Atomic index storage method based on bitmap summary model

An index and atomic technology, applied in the field of atomic index storage method and system based on the bitmap summary model, can solve the problems of low countdistinct efficiency, data skew of large data volume, inability to support business queries, etc., and achieve low storage and calculation efficiency , reduce storage size, and improve aggregation query efficiency

Active Publication Date: 2020-06-02
SUNING CLOUD COMPUTING CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] 1) The dimensions of the front-end query are flexible and changeable. On the one hand, there are too many dimension combinations, and the pre-calculation consumes a lot of resources (the background needs to calculate once for each dimension combination scenario). The summary table cannot support business queries and must be recalculated for new dimension combinations;
[0005] 2) The use of count distinct is inefficient, and it is very easy to generate data skew for large amounts of data, resulting in long-tail tasks

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Atomic index storage method based on bitmap summary model
  • Atomic index storage method based on bitmap summary model
  • Atomic index storage method based on bitmap summary model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

[0033] Such as figure 1 As shown, the present invention is based on the atomic index storage method of the bitmap summary model, and converts the atomic index into accumulative object storage, including the following steps:

[0034] Step 1, initialize the bitmap optimizer (BitSetOptimizer), including the element group object (ElementGroup) and the bit set group object (BitSetGroup), which are used to group and store the surrogate key digital indicators of atomic indicators;

[0035] BitSetOptimizer data structure such as figure 2 As shown, the ElementGroup object and th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an atomic index storage method and system based on a bitmap summary model, and the method comprises the steps that a bitmap optimizer carries out grouping storage of proxy keydigital indexes of atomic indexes, and comprises an element group object and a bit set group object; a digital coding module performs digital coding on the atomic indexes to be stored to generate proxy key digital indexes and corresponding group numbers of the atomic indexes; and a matching storage module respectively matches the atomic indexes to be stored with the bit set group object and the element group object of the bitmap optimizer, and stores proxy key digital indexes of the atomic indexes. The number of data entries and the storage size can be reduced, and the aggregation query efficiency is improved.

Description

technical field [0001] The invention relates to the field of information processing, in particular to an atomic index storage method and system based on a bitmap summary model. Background technique [0002] In the indicator system of Data Warehouse, there are atomic indicators (indicators that cannot be split further) that do not support cumulative aggregation, such as the number of visitors, number of members and other indicators. In the process of data aggregation, it is necessary to deduplicate based on visitor ID and member ID. Counted, deduplicated summary fact tables cannot support higher aggregations. [0003] The current data warehouse design deduplication index summary table generally uses a pre-calculation method, based on the fine-grained (including visitor ID, member ID dimension granularity) fact table and front-end display dimension combination, pre-calculates fixed dimension combinations, and directly generates front-end display. As a result, this scheme has ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/242G06F16/2455
CPCG06F16/2237G06F16/244G06F16/24556
Inventor 彭虎刘洋傅尚强施斌孙迁
Owner SUNING CLOUD COMPUTING CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap