Large-index quick splitting method based on Lucene

Abstract

The invention discloses a large-index quick splitting method based on Lucene, which comprises the following steps of: marking on a current newly-built index fragment directory by utilizing a soft connection principle of a file under Linux, and marking an original index file storage position pointed by a current file; deleting half of specified index data on the current newly-built index fragment and deleting the other half of opposite data on the other index fragment by means of the delta form characteristic of Lucene, so as to finish the process of splitting the index file into two parts fromone part; after index splitting is completed, determining a storage directory located by the current index data according to a deletion condition used during deletion operation, and re-storing subsequent data; according to the method provided by the invention, no extra copy overhead is needed in the splitting process, the efficiency of deleting the specified index data is high, and the index splitting process is accelerated; after the index splitting process is completed, the subsequent data storage index data positioning rule is related to the deletion condition of the deletion operation, noadditional algorithm is needed, and simplicity, convenience and quickness are achieved.

Description

technical field [0001] The invention relates to the technical field of file indexing, in particular to a method for quickly splitting large indexes based on Lucene. Background technique [0002] With the advent of the big data era, the amount of data is growing explosively. After the data is indexed at the time of storage, the retrieval performance of the data is greatly improved. Unfortunately, there is also a price to pay for indexing tables. First of all, the establishment of an index needs to occupy physical space. When there is more and more data, the index file will become larger and larger; secondly, it will take time to create and maintain the index, and this time will increase with the increase of the amount of data; When the data is added, deleted, and modified, the index must also be dynamically maintained. The more data, the larger the index file, and the lower the data maintenance efficiency. [0003] If there is only one index file or the number of shards in...

AI Technical Summary

Problems solved by technology

However, this process will be very time-consuming when the amount of data is too large, and if the original data is modified during the splitting process, it may cause data loss. Certain measures are required to ensure data security and novelty

[0005] Disadvantages of the existing technology: 1. The existing index splitting technology is to directly copy the index data to the new shard. When the amount of data is large, because of an extra copy, the Expensive and unnecessary

2. After setting the new number of shards, it may be necessary to rearrange all the data. If the amount of data is large, it will be very time-consuming

3. The existing index splitting technology needs to use a certain algorithm for index fragmentation to complete the positioning. The number of fragments is part of the algorithm, and the cost of modifying the number of fragments is very expensive

4. The growth of project data is unpredictable, and it is difficult to set the exact number of fragments

5. If sharding is performed during use, the sharding process will be very long. If the original data is modified during the splitting process, these modifications may be lost

6. If the shard is locked before the split and cannot be modified again, it cannot be modified until the split is completed, which will cause a large number of abnormal requests to the calling server because the split process is too long

Images