Resource management method and device for data warehouse system

Abstract

The embodiment of the invention discloses a resource management method and device for a data warehouse system. The method comprises the steps that in a data warehouse system, data files are distributed and stored in all buckets (storage spaces) of storage services, all the buckets are mapped to all computing nodes respectively, and all the computing nodes store metadata information of the data files of the mapped buckets and are responsible for computing tasks of the mapped buckets; in response to a trigger event for adjusting the computing resources, according to a mapping relation between the bucket and the computing nodes before the computing resources are adjusted and a mapping relation between the bucket and the computing nodes after the computing resources are adjusted, determining that the bucket where the mapped computing nodes are changed is used as a bucket to be migrated, and taking the computing node to which the bucket to be migrated is mapped before the computing resources are adjusted as a source computing node; the computing node to which the computing resource is mapped after being adjusted is used as a target computing node; and migrating metadata corresponding to the to-be-migrated bucket from the source computing node to the target computing node. According to the invention, the capacity expansion and contraction duration of the cloud data warehouse can be effectively shortened.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular, to a resource management method and device for a data warehouse. Background technique [0002] A data warehouse is a system that brings data from various data sources into storage and prepares it for subsequent quick retrieval. Data warehouses usually contain structured or semi-structured data extracted from various data sources. Data warehouses can be deployed locally, but due to hardware cost and scalability considerations, more and more users deploy data warehouses in the cloud. [0003] In most scenarios using cloud data warehouses, the time-sharing flexibility of the data warehouse is highly valued, which requires the adjustment of computing resources to be as fast as possible and to have as little impact on read and write operations as possible. The adjustment of computing resources in traditional cloud data warehouses (such as the expansion and contraction ...

AI Technical Summary

Problems solved by technology

Consistent hashing is based on the hash of the data file name, and then maps to the nearest computing node in the ring space, which causes different tables in the data warehouse to correspond to different computing nodes. Data transmission between, resulting in a waste of system resources

[0006] The data warehouse system AWS Redshif uses distributed hashing to map data slices to computing nodes. However, when computing nodes expand and shrink, data slices are remapped, and the divided data slices are migrated. This is a time-consuming process. time-consuming process, the expansion performance is poor

Images