Data pre-aggregation method and system, calculation device and storage medium

A pre-aggregation and data technology, applied in the field of big data, can solve the problems of large storage space, high management cost, and inability to quickly retrieve large-scale data, and achieve the effect of small storage space and low management cost.

Active Publication Date: 2020-05-01
HANGZHOU YITU MEDIAL TECH CO LTD
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to solve the problems in the prior art that data preprocessing requires pre-designed rules, which can only address the determined query requirements, cannot perform fast retrieval of large batches of data, occupies a large storage space, and has high management costs.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data pre-aggregation method and system, calculation device and storage medium
  • Data pre-aggregation method and system, calculation device and storage medium
  • Data pre-aggregation method and system, calculation device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] The implementation of the present invention will be illustrated by specific specific examples below, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention will be presented in conjunction with a preferred embodiment, it does not mean that the features of the invention are limited to this embodiment. On the contrary, the purpose of introducing the invention in conjunction with the embodiments is to cover other options or modifications that may be extended based on the claims of the present invention. The following description contains numerous specific details in order to provide a thorough understanding of the present invention. The invention may also be practiced without these details. Also, some specific details will be omitted from the description in order to avoid obscuring or obscuring the gist of the present invent...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data pre-aggregation method. The method comprises the following steps: setting a data pre-aggregation period and a high-frequency threshold; analyzing the query statement toobtain corresponding query information; recording a query log according to the query information, wherein the query log at least comprises a target table, an operation type and query time corresponding to the query information; analyzing the query log, and generating a pre-polymerization table in combination with the period and the high-frequency threshold value. Automatic dynamic adjustment can be realized, the management cost is low, and the occupied storage space is small. The invention further provides a data pre-aggregation system, calculation equipment and a storage medium.

Description

technical field [0001] The present invention relates to the field of big data, in particular to a data pre-aggregation method, system, computing device and storage medium. Background technique [0002] In the process of data warehouse construction, Apache Spark and Apache Presto are fast computing execution engines in the process of large-scale data processing. In order to speed up data query, data in multiple dimensions is usually aggregated according to the data query requirements of the business to form some large and wide tables. The data of which dimensions are aggregated depends on the business in principle. At the same time, it is not that the more aggregations, the better. The aggregation of dimensions can speed up the query, but it also consumes more storage space. Especially in complex data warehouse construction scenarios such as medical care and government affairs, there are other challenges, such as: containing thousands of tables, the design of the data model...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/242G06F16/2455
CPCG06F16/244G06F16/24556Y02D10/00
Inventor 郑永升石磊石权
Owner HANGZHOU YITU MEDIAL TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap