HTAP-oriented distributed database intelligent hybrid storage method

A hybrid storage and database technology, applied in the field of HTAP-oriented distributed database intelligent hybrid storage, can solve problems such as many iterations, difficult algorithm control, and difficult data set convergence.

Active Publication Date: 2019-08-20
UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF13 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

3. Not suitable for convex sample sets, because convex data sets are more difficult to converge
4. The selection of the initial value of each cluster center will affect the

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • HTAP-oriented distributed database intelligent hybrid storage method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0031] Such as figure 1 As shown, a HTAP-oriented distributed database intelligent hybrid storage method, the storage method includes:

[0032] Obtain the data in the data source through the data import system and store it in the storage engine of the HTAP database. The storage engine is composed of multiple storage nodes, and the data of each storage node is stored in the form of column family;

[0033] According to OLAP and OLTP business requests, the central node adopts the density-based clustering and partitioning algorithm to optimize and reorganize the data layout in the storage engine. With the continuous reorganization of the data layout, the number of column families and the number of columns in the The optimal layout obtained by the algorithm is constantly changing, and then the optimal data layout is obtained.

[0034] Such as figure 1 Shown is the system architecture, and the specific process of the inventive method is as follows:

[0035] (1) The data import sy...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an HTAP-oriented distributed database intelligent hybrid storage method. The method comprises the steps that data in a data source are obtained through a data import system andstored in a storage engine of an HTAP database, the storage engine is composed of a plurality of storage nodes, and the data of each storage node are stored in the data organization format of a column family; and the central node performs optimized recombination analysis on the data layout in the storage engine by adopting a density-based clustering partition algorithm according to the working load conditions of the historical OLAP and OLTP, so as to obtain the optimal data layout. According to the invention, the data in the HTAP database is organized in a column family manner; and the data in the column group is dynamically adjusted according to the optimal storage layout calculated by the central node, and the optimal storage layout is obtained by a density-based clustering algorithm through a clustering result, so that the columns with equivalent access frequencies belong to the same column group, i.e., the columns which are frequently accessed belong to the same column group.

Description

technical field [0001] The invention relates to the technical field of dynamically reorganizing the data layout of the storage engine by analyzing the workload of historical business and recent business through machine learning in the business scenario of HTAP, and specifically relates to an intelligent hybrid storage method for HTAP-oriented distributed databases . Background technique [0002] HTAP database is a distributed database product that supports both online transaction processing (OLTP) and online analytical processing (OLAP). Due to the very different characteristics of OLAP and OLTP systems, the data in the storage engine is stored in the form of rows. It is friendly to OLTP, and storage in the form of columns is more friendly to OLAP. However, if OLAP for efficient query and OLTP with high real-time requirements are better supported at the same time, then the data organization format of the storage engine plays a vital role. [0003] Currently, peloton storag...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22G06F16/27G06F9/50
CPCG06F16/221G06F16/278G06F9/5083
Inventor 段翰聪刘长红姚入榕闵革勇梁戈
Owner UNIV OF ELECTRONICS SCI & TECH OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap