Data index management method and device

Abstract

Embodiments of the invention disclose a data index management method and device, relates to the technical field of computers, and aims at decreasing a process of obtaining data through accessing each cluster node in an index creation process and then improving the index creation speed. The method comprises the following steps of: extracting original data from a data source and pushing the original data to a data warehouse tool list; decomposing data in the data warehouse tool list into an appointed quantity of data fragments according to a preset fragmentation rule, and distributing the data corresponding same identification information to a same data fragment; establishing an index creation tasks for each data fragment and guiding each data fragment into a corresponding index fragment. The method and device are suitable for rapidly creating indexes.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data index management method and device. Background technique [0002] With the development of computer technology, the amount of data that needs to be processed in real time has increased dramatically. In order to improve the efficiency of data query and calculation, indexing is mainly used to optimize the query and access mechanism for massive data. At present, there are two main ways to create indexes. One is to create indexes through distributed applications; the other is to rely on the hadoop platform and create indexes through map-reduce tasks based on the hash fragmentation rules of the distributed solution itself. [0003] However, when using the current scheme to create an index of a large amount of data, the required computing resources are positively correlated with the amount of data, and the time required to create the index is relatively long, especially...

AI Technical Summary

Problems solved by technology

[0003] However, when using the current scheme to create an index of a large amount of data, the required computing resources are positively correlated with the amount of data, and the time required to create the index is relatively long, especially when the index data is processed and merged, the data in the database It takes multiple communication transfers to complete the creation process, and the larger the database cluster size, the more communication transfer times, and in industries involving big data, the continuous expansion of the database cluster size has become an inevitable trend, so usually In the pre-preparation process, a lot of hardware resources are allocated for the index creation process, which leads to the process of index creation occupying more computing resources and taking a long time, and the system efficiency is low

Images