A System for Optimizing Parallel Data Loading Performance of Array Database

Abstract

The invention discloses a system for optimizing the parallel data loading performance of array databases. The system comprises a monitoring engine (1), wherein the monitoring engine (1) is used for collecting monitoring information of a database cluster (2), returning the monitoring information to a FastLoad system component (3) and monitoring database nodes (7) in real time; the FastLoad system component (3) is used for carrying out distribution method-based data loading on to-be-loaded data on the basis of the monitoring information, partitioning and loading the data; and the database cluster (2) is used for carrying out the data loading. The system disclosed by the invention can meet the loading requirements of array data and is a scientific data management tool system capable of carrying out data parallel loading on large-scale array model-based data; and the performance of the system is 4-6 times of that of original data loading mechanisms of database systems.

Description

technical field [0001] The invention relates to a system, in particular to a system for optimizing the parallel data loading performance of an array database. Background technique [0002] In the era of big data, scientific data in many disciplines such as bioinformatics, meteorology or astronomical sciences has grown very rapidly. If you want to analyze and process these data, you need to import the data into the database first. As the amount of data increases, the performance of the data loading method becomes more and more important. [0003] SciDB is an open source scientific database system for scientific data management and analysis. It uses the array data (Array) model, mainly developed by Stonebraker and sponsored by Paradigm4. The original intention of its design is to solve scientific problems such as large amount of data and data heredity in scientific research. Different from traditional DBMS, benefiting from the array data model, SciDB can provide large-scale...

AI Technical Summary

Problems solved by technology

However, due to the traditional relational data loading strategies adopted by these data loading mechanisms, they are not suitable for quickly loading scientific data usually represented by array models into distributed parallel systems, especially in the data size and dimension of these scientific data When the value becomes very large, the performance degradation of the traditional data loading method will be more obvious

Images